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Preface 


The  U.S.  Army  does  not  currently  have  tools  to  leverage  empirical  demand  and  usage 
data  to  identify  existing  or  emerging  problems  with  part  or  process  quality.  Instead,  it 
relies  on  Product  Quality  Deficiency  Reports  (PQDRs)  and  subject  matter  experts 
(SMEs)  from  various  organizations  to  monitor  and  react  to  part  quality  problems. 

Because  of  the  scope  of  the  Anny’s  operations  (large  range  of  weapon  systems,  suppliers, 
and  parts)  and  the  lack  of  information  on  the  total  cost  of  part  deficiencies,  this  approach 
tends  toward  a  narrow  focus  either  on  very  expensive  parts,  such  as  transmissions,  rotor 
blades,  and  engines,  or  on  safety-critical  items.  The  current  approach  is  labor  intensive, 
and  it  does  not  prioritize  corrective  action  based  on  what  the  part  deficiency  is  costing. 

The  Army  asked  RAND  Arroyo  Center  to  develop  a  method  that  uses  readily 
available  data  sources  to  detect  potential  part  quality  problems  and  to  estimate  the 
potential  cost  of  the  problem  to  the  Army. 

This  report  on  the  cost  of  poor  part  quality  presents  the  results  of  the  first  year  of  a 
RAND  Arroyo  Center  study  sponsored  by  U.S.  Army  Materiel  Command  (AMC)  and  the 
Deputy  Chief  of  Staff,  G-4.  We  explore  the  feasibility  of  using  readily  available  demand 
and  end  item  maintenance  history  to  identify  potential  part  or  process  quality  issues  and 
estimate  their  associated  incremental  cost.  Because  of  the  availability  of  flight  hour  data, 
we  focus  our  study  on  aviation. 

This  research  was  sponsored  by  U.S.  Anny  Materiel  Command  (AMC)  and  the 
Deputy  Chief  of  Staff  G-4  and  conducted  within  RAND  Arroyo  Center’s  Military 
Logistics  Program.  RAND  Arroyo  Center,  part  of  the  RAND  Corporation,  is  a  federally 
funded  research  and  development  center  sponsored  by  the  United  States  Army. 

The  Project  Unique  Identification  Code  (PUIC)  for  the  project  that  produced  this 
document  is  HQD1 15822. 

For  more  information  on  RAND  Arroyo  Center,  contact  the  Director  of  Operations 
(telephone  310-393-0411,  extension  6419,  fax  310-451-6952;  email 
Marcy_Agmon@rand.org),  or  visit  Arroyo’s  web  site  at  http://www.rand.org/ard.html. 
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Summary 


This  report  presents  research  sponsored  by  U.S.  Anny  Materiel  Command  (AMC) 
and  the  Deputy  Chief  of  Staff,  G-4  to  explore  the  feasibility  of  using  readily  available 
demand  and  end  item  maintenance  history  to  identify  potential  issues  with  part  or  process 
quality  and  estimate  their  associated  incremental  costs. 

Currently,  the  Army  relies  on  Product  Quality  Deficiency  Reports  (PQDRs)  and 
subject  matter  experts  (SMEs)  from  various  organizations  to  monitor  and  react  to  part 
quality  problems.  Because  of  the  scope  of  the  Army’s  operations  (and  accordingly  its 
large  range  of  weapon  systems,  suppliers,  and  parts)  and  the  lack  of  information  on  the 
total  cost  of  part  deficiencies,  which  include  repair  and  maintenance,  inventory,  and 
disposal  costs  in  addition  to  the  procurement  cost,  this  approach  tends  to  focus  either  on 
very  expensive  parts,  such  as  transmissions,  rotor  blades,  and  engines,  or  on  safety- 
critical  items.  It  does  not  tend  to  focus  on  less-expensive  items,  which  may  be  total  cost 
drivers. 

The  purpose  of  this  analysis  is  threefold: 

•  First,  we  identify  three  part  usage  patterns  that  could  indicate  emerging  or 
existing  part  or  process  quality  problems. 

•  Second,  we  develop  and  test  two  methods  to  evaluate  the  performance  of 
thousands  of  parts  and  identify  potential  emerging  or  existing  issues.  The  methods 
rely  on  well-understood  statistical  tests  to  identify  reduced  reliability  and  to  group 
parts  for  comparison. 

•  Finally,  we  develop  a  cost  model  incorporating  estimates  of  increased 
maintenance  and  inventory  costs. 

This  approach  identifies  parts  for  further  analysis  based  on  the  estimated  total  system 
cost  of  poor  quality,  allowing  the  Army  to  focus  on  high-marginal-cost  items  regardless 
of  a  part’s  unit  price.  This  approach  is  similar  to  others  such  as  control  charts  and 
Weibull  analysis,  in  which  a  tool  identifies  potential  problems  leading  to  root  cause 
analysis  and  corrective  action. 

A  case  study  using  the  UH-60M  demonstrates  the  potential  for  this  methodology.  The 
value  of  this  approach  is  its  power  to  analyze  the  performance  of  thousands  of  parts,  and 
by  assigning  a  cost,  to  prioritize  a  list  of  parts  with  the  highest  potential  return  on 
investment. 
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1.  Introduction:  Managing  the  Cost  of  Poor  Quality 


Over  the  next  10  years,  the  Department  of  Defense  (DoD)  will  be  asked  to  reduce  its 
projected  spending  by  more  than  $450  billion.  Finding  a  way  to  implement  these  cuts  is  a 
critical  challenge  facing  the  Department  of  Defense.  In  his  first  major  policy  address, 
Secretary  of  Defense  Leon  Panetta  (Panetta,  2011)  said: 

In  this  fiscal  environment,  every  program,  every  contract  and  every  facility  will  be 
scrutinized  for  savings,  savings  that  won’t  reduce  readiness  or  our  ability  to 
perform  essential  missions  .  .  .  These  cuts  will  need  to  be  carefully  targeted  ...  to 
avoid  a  hollow  force,  to  ensure  that  we  maintain  a  robust  industrial  base,  and  to 
protect  the  new  military  capabilities  we  need  to  sustain  our  military  strength. 

This  research  can  help  the  U.S.  Anny  maintain  equipment  readiness  levels  in  an 
austere  budget  environment  by  controlling  the  incremental  maintenance  and  inventory 
costs,  as  well  as  the  increased  equipment  downtime,  created  by  poor  quality  processes 
and  parts. 

The  Relationship  Between  Quality,  Cost,  and  Readiness 

There  are  a  variety  of  reasons  why  a  part  may  fail  before  its  expected  life  limit, 
including  manufacturing  defects,  poor  part  design,  misuse,  or  incorrect  maintenance. 
These  factors  contribute  to  increased  costs  for  operations,  many  of  which  are  very 
difficult  to  quantify  or  are  unknown  and  unknowable.1  For  example: 

•  Flow  much  does  it  cost  when  a  part  fails  prematurely  in  the  field,  or  when  a 
production  line  is  stopped  or  slowed  due  to  poor-quality  parts? 

•  What  costs  are  generated  when  parts  are  incorrectly  diagnosed  due  to  outdated 
diagnostic  software? 

•  How  much  can  be  gained  through  improved  design  of  parts? 

•  What  is  the  impact  of  part  quality  deficiencies  on  equipment  readiness?2 

Currently,  the  Army  does  not  have  a  way  to  quantify  these  costs  and  therefore  does  not 
have  a  way  to  prioritize  quality  improvement  actions  based  on  their  potential  cost 


1  Unknown  and  unknowable  costs  related  to  poor  part  quality  include  decreased  availability  of  a  system  or 
reduced  confidence  in  a  system’s  operational  reliability. 

2 

Equipment  readiness  is  measured  by  the  product  of  the  time  required  to  order  and  repair  an  item  and  the 
failure  rate  (Peltz  et  ah,  2002) 
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avoidance.  Instead,  as  we  shall  discuss  in  Chapter  2,  current  quality  management  efforts 
are  heavily  dependent  on  Product  Quality  Deficiency  Reports  (PQDRs)  or  on  resource¬ 
intensive  analysis  of  maintenance  logbooks.  Because  the  current  approaches  do  not 
attempt  to  capture  costs  beyond  an  item’s  unit  price,  they  tend  to  focus  resources  on 
expensive  parts. 

We  broaden  this  perspective  by  estimating  costs  of  additional  depot  repair  workload 
generated  by  increased  failures  and  inventory  required  to  maintain  readiness,  thereby 
allowing  for  a  different — and,  we  propose,  better — prioritization  of  quality  improvement 
efforts. 

Background 

The  U.S.  Army  Materiel  Command  (AMC)  manages  weapon  system  procurement 
and  maintenance  for  the  Army.  The  AMC  mission  includes  procurement  of  spare  parts 
for  field  and  depot  level  maintenance,  modifications  and  modernization  of  existing  parts, 
and  recapitalization  of  equipment.  AMC  procures  millions  of  parts  and  services  from 
thousands  of  suppliers  each  year,  including  major  suppliers  such  as  organic  maintenance 
and  overhaul  depots  like  the  Corpus  Christi  Anny  Depot  (CCAD),  the  Defense  Logistics 
Agency  (DLA),  and  original  equipment  manufacturers  (OEMs).  In  turn,  these  major 
suppliers  procure  parts  and  services  from  many  sub-tier  suppliers,  with  no  visibility  of  the 
sub-tier  suppliers  by  the  Army.  It  is  not  uncommon  to  have  multiple  suppliers  for  the 
same  part,  or  changes  in  suppliers  as  contracts  expire  and  new  contracts  are  awarded.  The 
volume  of  supply,  lack  of  visibility,  multiple  suppliers,  and  changes  in  suppliers  are 
among  the  challenges  to  instituting  a  systematic  part  quality  management  process.  For  the 
Army,  the  cost  of  a  poor-quality  part  entering  the  system  manifests  itself  in  one  of  three 
ways: 

•  A  part  causes  a  failure,  which  compromises  safety  (parts  of  this  type  are 
categorized  as  critical  safety  items  (CSI). 

•  A  part  causes  a  failure  that  results  in  a  weapon  system  being  taken  out  of  service 
temporarily,  reducing  readiness  and  perhaps  compromising  the  ability  to  fulfill  a 
mission  objective/ 

•  (3)  Costs  rise  because  of  incremental  increases  in  maintenance  and  inventory 
level  needed  to  assure  readiness,  or  increased  inspection  and  removals  to  insure 
safety. 

The  first  two  cost  categories  are  the  direct  result  of  poor  part  quality  and  include  tangible 
costs,  such  as  the  cost  of  a  crashed  helicopter,  and  intangible  costs,  such  as  decreased 
confidence  among  warfighters  in  the  systems  on  which  they  rely.  The  third  category  is 
the  cost  of  mitigating  the  effects  of  the  first  two  by  investing  in  buffer  stocks  and 
inspections. 
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AMC  asked  RAND  Arroyo  Center  to  develop  a  method  that  would  allow  the  Army  to 
identify  the  costs  incurred  due  to  poor-quality  parts  and  help  it  identify  which  suppliers 
could  be  disproportionally  contributing  to  maintenance  costs. 

In  the  course  of  this  research,  it  became  evident  that  in  addition  to  the  manufacturing 
quality  of  the  part,  other  quality  factors  contribute  to  the  overall  cost  of  maintaining  a 
desired  level  of  readiness.  It  is  therefore  necessary  to  look  beyond  manufacturing  or 
repair  as  a  root  cause  for  part  quality  defects  and  costs. 

Study  Objective  and  Approach 

The  objective  of  this  research  is  to  develop  a  methodology  that  can  be  used  to 
systematically  examine  a  large  number  of  parts,  identify  parts  with  potential  quality 
deficiencies,  and  quantify  the  costs  of  those  deficiencies.  This  methodology  will  be  used 
to  identify  and  prioritize  corrective  actions  and  quality  improvement  initiatives.  There  is 
a  breadth  of  potential  users — from  depot  subject  matter  experts,  to  program  offices,  and 
to  item  managers  responsible  for  maintaining  a  supplier  base.  These  initiatives  could 
include,  but  are  not  limited  to: 

1 .  improving  technical  manuals  and  training  to  reduce  user-induced  failures; 

2.  working  with  suppliers  to  reduce  manufacturing  or  repair  defects; 

3.  identifying  opportunities  to  increase  reliability  by  redesigning  a  part;  and 

4.  prioritizing  quality  improvement  initiatives  for  problematic  parts. 

This  methodology  can  also  be  used  to  communicate  quality  problems  from  the  field  and 
produce  actionable  data  for  HQ  AMC. 

Our  approach  is  to  use  readily  available  part  supply  and  maintenance  data,  as  well  as 
statistically  based  tests,  to  identify  parts  that  are  exhibiting  decreasing  reliability  or  are 
less  reliable  than  similar  parts.  We  then  estimate  the  additional  maintenance  and 
inventory  costs  incurred  by  the  Army  because  of  the  identified  quality  deficiency.  The 
costs  are  based  on  a  six-month  projection  of  current  reliability  and  can  be  used  to 
prioritize  and  monitor  quality  improvement  initiatives. 

This  report  presents  the  development  and  application  of  Arroyo’s  Cost  of  Poor 
Quality  estimation  process  to  Army  Aviation.  We  present  a  case  study  using  the  UH- 
60M,  the  Black  Hawk  Mike  model. 


Organization  of  the  Document 

The  remainder  of  this  document  is  organized  in  five  additional  chapters: 

•  Chapter  2  describes  the  Army’s  current  system  for  identifying  and  managing  part 
quality  deficiencies. 

•  Chapter  3  provides  a  background  on  the  cost  of  quality. 
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•  Chapter  4  describes  our  methodology  for  identifying  part  quality  problems  and  for 
assigning  costs  for  part  quality  deficiencies. 

•  Chapter  5  presents  a  case  study  using  the  UH-60M  model. 

•  Chapter  6  presents  conclusions  and  recommendations. 
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2.  How  Does  the  Army  Currently  Monitor  Part  Quality 
Problems? 


There  are  various  organizations  responsible  in  one  way  or  another  for  assuring  Army 
parts  quality. 

•  The  RDECOM  organizations  such  as  Aviation  and  Missile  Research 
Development  and  Engineering  Center  (AMRDEC),  which  provide  engineering 
support  to  program  offices  and  conducts  supplier  audits. 

•  AMC/G-4  Continuous  Process  Improvement  (CPI)  is  a  recently  established  group 
responsible  for  setting  policy  on  quality  for  AMC  and  driving  continuous 
improvement  throughout  the  command. 

•  The  Defense  Contract  Management  Agency  (DCMA)  is  responsible  for 
overseeing  contract  requirements  and  assuring  that  items  conform  to 
specifications. 

•  The  Life  Cycle  Management  Centers  (LCMCs)  are  responsible  for  managing 
maintenance  activities,  including  logistics,  materiel  and  supply  chain 
management,  and  strategic  sourcing. 

These  organizations  have  limited  tools  to  assess  part  quality.  To  date,  Product  Quality 
Deficiency  Reports  (PQDRs)  and  logbook  data  are  used  for  the  task.  Below  we  describe 
these  data  sources  and  how  the  listed  organizations  apply  PQDR  data  to  measuring 
quality. 

Product  Quality  Deficiency  Reports 

The  primary  method  used  to  identity  and  track  part  quality  problems  in  the  field  is  the 
PQDR.  A  PQDR  is  to  be  filed  on  any  item  with  “any  defect,  nonconforming  condition,  or 
premature  equipment  failure  indicating  deficiencies  in  design,  specification,  material, 
manufacturing,  and  workmanship  or  for  deficiencies  in  major  weapon  systems, 
secondary/consumable/repairable  items,  spare  and  repair  parts,  Gov-owned  products,  and 
Gov-furnished  property”  (Product  Quality  Deficiency  Report  Process,  2008).  The  PQDR 
process  is  a  DoD  standard  used  by  all  services  (DLA,  1993). 

PQDRs  are  classified  according  to  their  level  of  severity.  Category  I  (CATI)  PQDRs 
are  deficiencies  which  may: 

•  lead  to  death,  injury,  severe  illness; 

•  cause  major  damage  to  a  weapon  system;  or 

•  critically  restrict  combat  readiness  capabilities. 
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Category  II  PQDRs  encompass  all  deficiencies  that  are  not  Category  I.  Once 
discovered,  CATI  PQDRs  must  be  reported  by  the  originating  organization  to  the  PQDR 
screening  point  within  24  hours.  The  screening  point  validates  the  PQDR  and,  if  valid, 
enters  the  report  into  the  PQDR  data  system  and  forwards  it  to  the  action  point.  For  CATI 
PQDRs,  the  screening  point  has  24  hours  to  process  the  report  and  forward  it  to  the  action 
point  for  further  investigation.  If  severe,  CATI  PQDRs  can  lead  to  grounding  or  other 
severe  restrictions  on  the  use  of  a  weapon  system  until  the  root  cause  of  the  deficiency  is 
uncovered. 

In  addition  to  quickly  communicating  potentially  critical  quality  problems,  PQDRs 
serve  another  important  function.  If  a  part  fails  in  the  field,  the  customer  will  receive  a 
credit  for  the  part  if  the  failure  was  determined  to  be  a  valid  product  quality  deficiency. 
Therefore,  there  is  an  incentive  for  units  to  file  PQDRs  for  both  safety  and  financial 
reasons. 

In  theory,  a  part  that  fails  in  the  field  for  other  than  normal  wear  will  generate  a 
PQDR.  In  practice,  there  is  no  mechanism  to  ensure  that  all  deficiencies  are  captured. 
Interviews  with  subject  matter  experts  (RAND,  2011)  indicate  that  PQDRs  are  not  used 
to  report  all  instances  of  deficiencies.  Several  reasons  were  given  for  this: 

1 .  The  PQDR  process  can  be  time  consuming,  and  if  the  deficiency  is  not  a  CATI, 
there  is  little  incentive  for  a  unit  to  file  a  report  unless  the  part  is  expensive  and 
the  unit  wants  a  credit. 

2.  When  the  deficiency  is  well  known,  i.e.,  other  PQDRs  have  been  filed  and  are 
under  investigation,  the  originating  unit  may  be  asked  not  to  file  an  additional 
PQDR. 

Because  of  this,  PQDR  reporting  is  biased  toward  deficiencies  for  more  expensive  items, 
or  for  safety-critical  items.  Table  2.1  shows  how  the  distribution  of  PQDRs  filed  in  2010 
for  Anny  Aviation  National  Stock  Numbers  (NSNs)  tended  to  be  more  representative  of 
NSNs  valued  at  $40,000  or  more.  Column  1  of  Table  2.1  categorizes  NSNs  by  unit  price; 
Column  2  and  Column  3  show  the  count  and  the  percent  of  the  total  of  all  critical  NSNs3 
for  all  weapon  systems  as  recorded  in  the  Equipment  Downtime  Analyzer  (EDA)  data 
from  2004  to  2010.  Over  89  percent  of  the  parts  identified  as  critical  are  valued  under 
$1,000.  The  distribution  changes  slightly  as  we  limit  our  selection  to  parts  used  by  Army 
Aviation,  Column  4. 4  Column  7  shows  the  percentage  distribution  by  price  category  of 
aviation  NSNs  that  also  received  a  PQDR.  The  takeaway  is  that  only  2  percent  of  the 
NSNs  used  to  maintain  aviation  assets  were  valued  at  $40,000  or  more,  and  they 


3 

“Critical”  means  that  the  NSN  appeared  on  a  job  order  for  the  repair  of  an  end  item  that  was  down  for 
maintenance  for  more  than  one  day. 

4  NSNs  used  on  repair  of  UH-60,  OH-58,  CH-47,  AH-64. 
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generated  16  percent  of  the  PQDRs.  While  less  expensive  NSNs  generate  PQDRs,  they 
do  so  disproportionately  to  the  number  of  NSNs  in  the  price  category.  While  the  price  of 
the  parts  on  a  weapon  system  varies  greatly,  the  cost  of  having  a  piece  of  equipment  out 
of  service  is  the  same  regardless  of  the  price  of  the  part  downing  the  equipment.  The  bias 
toward  more  expensive  parts  limits  opportunities  for  cost  avoidance  and  improvement  of 
reliability. 


Table  2.1.  Distribution  of  PQDRs  by  Price  Category 


Price 

Number  of 
NSNs  Critical 
(All  systems) 

Percent 
(All  systems)* 

Number  of 
NSNs 
Critical 
(Aviation) 

Percent 

Aviation* 

Number 
NSNs  with 
PQDRs 
(2010,  Army 
Aviation) 

Percent 
PQDR  Army 
Aviation 
2010 

0-100 

90006 

66% 

11312 

58% 

202 

13% 

101-1000 

30957 

23% 

4346 

22% 

310 

19% 

1001-10K 

11661 

9% 

2749 

14% 

535 

33% 

10K-20K 

1316 

1% 

399 

2% 

151 

9% 

20K-30K 

564 

0% 

202 

1% 

83 

5% 

30K-40K 

320 

0% 

127 

1% 

62 

4% 

>40K 

994 

1% 

393 

2% 

264 

16% 

Analysis  by  Subject  Matter  Experts 

PQDRs  are  not  the  only  means  used  to  report  part  quality  problems.  Program 
Managers  within  the  Program  Executive  Office  monitor  part  problems  reported  from  the 
field  and  work  with  suppliers  and  field  maintainers  to  troubleshoot  part  reliability  issues. 
Interviews  conducted  with  the  Reliability  and  Maintainability  (RAM)  Office  for  the 
UH-60  program  indicated  that  the  program  offices  use  subject  matter  expertise  and  strong 
ties  to  the  maintenance  community  to  analyze  and  identify  potential  problems.  Subject 
matter  experts  (SMEs)  and  supporting  offices  read  thousands  of  maintenance  logs, 
looking  for  indications  of  emerging  or  ongoing  part  quality  issues.5  Multiple  entries  in  the 
electronic  maintenance  logs,  across  the  entire  fleet  of  aircraft,  are  used  to  highlight 


5  The  logbook  entries  are  a  series  of  free-form  narratives  that  capture  the  mechanic’s  description  of  the 
maintenance  action.  The  logbook  data  does  not  use  any  codes  that  can  be  queried,  so  manual  interpretation 
of  the  individual  entries  is  needed  to  analyze  this  data  source.  The  technical,  shorthand  nature  of  the 
reporting  requires  knowledgeable  interpreters,  such  as  former  crew  chiefs,  to  translate  the  narratives  into 
meaningful  data  for  analysis.  Accordingly,  only  10  percent  of  maintenance  logs  are  reviewed  and  scored, 
i.e.,  cleansed  enough  for  analysis  (RAND  research  interview  with  AMRDEC,  2011). 
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potential  problems.  SMEs  then  contact  field  and  depot  maintenance  offices  to  investigate 
possible  causes  for  an  increased  occurrence  of  maintenance  issues. 

Although  it  is  effective,  this  process  has  a  couple  of  drawbacks.  First,  it  is  very  labor 
intensive.  The  Unit  Level  Logistics  System  Aviation  Enhanced  (ULLS-A(E))  electronic 
logbook  is  used  by  field  level  mechanics  to  record  all  maintenance  activities  performed 
on  an  aircraft.  Table  2.2  shows  a  sample  from  a  list  of  ULLS-A(E)  logbook  entries.  The 
information  is  not  tracked  by  serial  number  or  NSN,  which  requires  SMEs  to  use  the 
narrative  fields  to  identify  the  type  of  component  being  discussed  and  the  cause  of  its 
failure.  In  its  raw  state,  the  ULLS-A(E)  data  is  not  designed  to  facilitate  the  type  of  data 
analysis  leading  to  the  detection  of  a  trend. 


Table  2.2.  Sample  ULLS-A(E)  Data  Entry 


Fault  Detected 

Action 

STAB  FAILED  DURING  HYD  LEAK  TEST  AND  TAIL  ROTOR 

SERVO  CHECK.  ADDITIONALLY,  APU  GEN  FAILED  DURING 

SECOND  TRY. 

REPLACED  GCU. 

#1  GEN  FAILED  IN  FLIGHT.  AN  ATTEMPT  WAS  MADE  TO 

RESET  THE  GENERATOR  THAT  DID  NOT  WORK.  UPON 

LANDING  AND  SHUT  DOWN  THE  #1  GEN  CAUTION  WENT 

OUT  AND  THE  GEN  CAME  BACK  ON  LINE. 

REPLACED  #1  GCU  S/N  1493FM 

MOC  REQ'D  FOR  THE  REPLACEMENT  OF  #1  GCU 

MOC  OK 

MOC  REQ'D  FOR  REPLACEMENT  OF  APU  GCU 

MOC  OK 

#2  GEN  LIGHT  WILL  NOT  GO  OFF  WITH  GEN  SWITCH  IN  ON 

POSITION 

INSTALLED  #2  GCU 

STAB  FAILS  DURING  HYD  LEAK  TEST  AND  TAIL  ROTOR 

SERVO  CHECK. 

REPLACED  GCU  WITH  GCU  SERIAL  #  2539F. 

Second,  SMEs  are  located  within  program  offices  and  devote  their  attention  to  one 
weapon  system.  Therefore,  they  are  apt  to  miss  quality  problems  that  cross  weapon 
systems,  such  as  when  a  part  is  used  on  a  CH-47  as  well  as  a  UH-60.  Because  the  UH-60 
is  flown  at  a  higher  rate  than  the  CH-47,  parts  may  start  exhibiting  problems  on  the 
UH-60  fleet  first,  resulting  in  the  possibility  that  the  CH-47  Program  Office  would  not  be 
aware  of  this  early  indication  of  potential  problems.  Our  methodology  overcomes  this 
issue  by  systematically  examining  similar  parts  across  weapon  systems. 

Communicating  to  Top  Management 

The  PQDR  reports  and  quality  monitoring  by  SMEs  do  not  by  themselves  constitute  a 
quality  management  process.  In  order  to  prioritize  problems  and  direct  resources,  top 
management  needs  a  standardized  method  for  receiving  reports  on  quality  issues  from  the 
field.  The  current  process  used  by  AMC  to  monitor  part  quality  problems  in  the  field  is  a 
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Weekly  Production  Update  (WPU)  presented  by  the  Life  Cycle  Management  Commands 
(LCMCs).  Each  LCMC  (TACOM,  AMCOM,  CECOM,  and  JM&L  COM)6  takes  turns 
presenting  briefs  to  senior  AMC  management  on  key  business  processes  (one  report 
every  six  weeks.)  The  report  on  quality  problems  is  focused  on  PQDRs:  specifically, 
senior  AMC  managers  are  briefed  on  the  top  five  open  PQDRs,  by  dollar  value  and  by 
age.  Figure  2. 1  shows  an  example  from  a  recent  WPU  report  presented  by  AMCOM. 

Figure  2.1.  AMCOM  Quality  Report  Presented  at  Weekly  Production  Update 


TOP  5  Items  with  PQDRs  by  Value  ($M) 

RCN# 

(‘Oldest  PQDR) 

Nomenclature 

l/E 

SOS 

NIIN 

CAT 

Qty 

Def 

Unit  Cost 

Total  # 
PQDR 
s 

End  Item 

Value 

($M) 

Age* 

(Days 

) 

W25N7V-10- 

0167/68 

Engine,  A/C  Turboshaft 

1 

AMC 

015031701 

I/ll 

17 

706862 

17 

UH-60/ 

AH-64 

11.73 

279 

077272-10-0021 

Engine,  A/C  Turboshaft 

E 

AMC 

014585361 

II 

3 

941364 

3 

CH-47 

2.824 

286 

WT4XBL-10- 

0006/7 

Cold  Section  Module 

E 

AMC 

012844013 

II 

7 

329553 

7 

UH-60/ AH- 
64 

2.286 

143 

W81JMJ-1 0-0011 

Transmission, 

Mechanical 

E 

AMC 

015209744 

I/ll 

5 

401667 

5 

AH-64D 

1.19 

166 

W81HL5-1 0-0014 

Transmission, 

Mechanical 

E 

AMC 

015209742 

I/ll 

4 

401667 

4 

AH-64D 

1.581 

145 

TOP  5  PQDRs  by  Age  (Days) 

RCN# 

Nomenclature 

l/E 

SOS 

NIIN 

CAT 

Unit  Cost 
Def 

End  Item 

Value 

($M) 

Age 

(Days) 

W45N7V-09-0274 

Spring,  Helical 

1 

SMS 

011291953 

II 

42 

24.16 

CH-47D 

.0010 

577 

W25G1Q-1 0-0071 

Charger,  Battery 

1 

AMC 

113377422 

II 

3 

14,712 

AGPU 

.044 

395 

W90HXE-1 0-0003 

Accumulator,  Hydraulic 

1 

AMC 

012224316 

II 

3 

9,326 

UH-60A 

.028 

385 

W45N7V-10-0015 

Altimeter,  Encoder 

1 

AMC 

011769314 

II 

2 

4,912 

UH-60L 

.010 

379 

W912U7-1 0-0003 

Nozzle,  Turbine  Engine 

1 

AMC 

014537890 

II 

1 

24,752 

CH-47F 

.025 

377 

The  WPU  report  provides  a  snapshot  in  time  of  top  quality  drivers.  Including  a  report 
on  the  top  five  PQDRs  (in  terms  of  both  value  and  days  open)  is  meant  to  increase  the 
visibility  of  unresolved  issues.7  However,  because  there  is  no  trend  data,  it  is  hard  to 
determine  whether  quality  issues  are  improving  or  worsening.  The  top  five  by  dollar 
value  is  detennined  by  multiplying  the  current  price  of  the  item  by  the  number  of  defects 
reported  in  the  PQDRs.  Therefore,  it  will  generate  a  list  of  the  most  expensive  NSNs  or 
less  expensive  items  that  have  had  a  very  large  number  of  quality  deficiencies.  This  list 
will  not  include  items  that  did  not  receive  PQDRs,  and  it  will  exclude  chronic  problems 


6  Tank  and  Automotive  LCMC,  Aviation  and  Missile  LCMC,  Communications-Electronics  LCMC,  and  the 
Joint  Munitions  and  Lethality  LCMC. 

7  The  expectation  is  that  by  discussing  the  PQDR  during  the  WPU,  action  will  be  taken  fix  the  problem.  If 
the  same  PQDR  shows  up  in  successive  WPUs,  the  LCMC  commander  is  asked  to  explain  why.  When  the 
PQDR  is  closed,  it  drops  off  the  WPU  report. 
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that  do  not  reach  the  top  dollar  value  criteria  at  that  point  in  time.  Similarly,  the  top  five 
by  days  list  is  meant  to  help  management  identify  PQDRs  that  have  not  been  closed,  i.e., 
a  determination  on  the  cause  of  the  quality  deficiency  has  not  been  completed. 

The  Continuous  Process  Improvement  (CPI)  Division  recognizes  some  of  the 
shortcomings  of  the  current  process.  They  plan  to  expand  the  reporting  of  PQDRs  and 
require  more  rigor  in  the  WPU  reporting.  We  propose  that  the  Army  move  toward  an 
enterprise  view  of  quality  issues  and  broaden  the  scope  of  “quality  deficiencies”  beyond 
PQDRs.  Since  one  of  the  most  effective  ways  to  focus  attention  on  a  problem  is  to  attach 
a  dollar  value  to  the  cost  generated,  our  approach  incorporates  a  cost  estimating  function, 
along  with  two  complementary  methods  for  identifying  part  quality  issues. 
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3.  Estimating  the  Cost  of  Quality 


Introduction  to  the  Cost  of  Quality 

The  term  “cost  of  quality”  (COQ)  is  somewhat  contentious.  Some  have  interpreted  it 
to  mean  the  costs  of  quality  improvement  programs.  However,  a  generally  accepted 
definition  of  the  COQ  is  “the  sum  of  conformance  plus  nonconformance  costs,  where 
cost  of  conformance  is  the  price  paid  for  prevention  of  poor  quality,  and  cost  of  non¬ 
conformance  is  the  cost  of  poor  quality  caused  by  product  and  service  failure” 
(Schiffauerova,  2006). 

The  cost  of  conformance  can  include 

•  operations  and  process  validation, 

•  quality  system  audits,  and 

•  maintenance  of  equipment. 

The  cost  of  nonconformance  can  include 

•  customer  dissatisfaction  leading  to  lost  sales  (and  revenue), 

•  compromised  employee  morale  due  to  persistent  quality  issues, 

•  increased  scrap  and  rework  costs  because  of  defective  raw  materials, 
manufacturing  defects,  or  inadequate  maintenance  procedures,  and 

•  expedited  shipping  costs  due  to  poor  order  fulfillment,  etc.* 8 

The  literature  on  COQ  refers  to  the  difficulty  of  estimating  these  costs  (Campanella, 
1999).  In  part,  this  difficulty  arises  because  accounting  and  financial  systems  are  not 
designed  to  capture  these  types  of  costs.  However,  the  literature  and  case  studies 
emphasize  the  importance  of  estimating  the  cost  of  quality  as  a  means  of  driving  change 
throughout  an  organization  and  for  helping  management  prioritize  and  select  quality 
improvement  activities.  For  example,  by  tracking  the  cost  of  quality,  CRC  Industries  was 
able  to  reduce  “failure  dollars”9  from  0.70  percent  of  sales  to  0.21  percent  of  sales  over  a 
nine-year  period.  They  did  this  by  collecting  data  on  four  categories  of  failure  dollars: 

•  Internal  quality  incidents:  costs  related  to  correcting  product  defects. 

•  Scrap/waste,  includes  material  scrapped  due  to  defects. 


g 

See  Appendix  B  of  Principles  of  Quality  Costs  (Campanella,  1999). 

9 

Failure  dollars  are  the  money  spent  because  products  and  services  did  not  meet  customer  requirements. 
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•  Customer  complaints/recalls:  all  costs  involved  in  resolving  a  customer 
complaint  or  recall,  including  claims,  shipping  costs,  and  labor  costs. 

•  Product  destroyed  in  field/warranty:  the  cost  of  deductions  CRC  distributors 
take  for  product  returned  by  customers.  (Donovan,  2006) 

The  point  illustrated  by  the  CRC  case,  and  supported  by  the  COQ  literature,  is  that 
tracking  and  improving  a  limited  number  of  the  many  potential  COQ  categories  can 
greatly  reduce  overall  operating  costs. 

Estimating  the  Cost  of  Poor  Part  Quality  from  the  Army  Perspective 

By  estimating  the  cost  of  poor  part  quality  and  incorporating  it  into  the  metrics 
reported  by  the  LCMCs,  the  PMs,  and  AMRDEC  and  by  prioritizing  Lean  Six  Sigma 
activities  based  on  projected  cost  savings,  AMC  can  (1)  send  a  message  to  the 
subordinate  commands  that  measureable  cost  reductions  must  be  achieved;  (2)  link  poor 
quality  and  reduced  reliability  to  cost,  emphasizing  the  importance  of  improving  quality 
to  the  overall  objective  of  reducing  costs,  and  (3)  properly  prioritize  and  allocate  scarce 
resources  where  they  have  the  greatest  impact  on  cost.  The  Army  faces  a  daunting 
challenge  of  systematically  identifying  parts  or  processes  that  have  a  quality  issue  and 
associating  a  cost  to  this  quality  issue  for  proper  prioritization  and  allocation  of  resources. 

Potential  for  Reducing  the  Costs  of  Poor  Part  Quality 

Unlike  CRC,  the  Army  does  not  generate  revenue  in  the  conventional  meaning  of  the 
word.  Instead,  the  DoD,  under  the  provisions  of  Title  10,  established  the  Army  Working 
Capital  Fund  (AWCF)  to  provide  inventories  of  supplies  and  industrial  activities  that 
provide  common  services,  such  as  repair,  manufacturing,  or  remanufacturing.  The  AWCF 
pays  for  inventory  and  depot  repair  under  a  revolving  fund  concept  whose  goal  is  to 
break  even  by  returning  any  monetary  gains  to  appropriated  fund  customers  through 
lower  rates  or  collecting  any  monetary  losses  from  customers  through  higher  rates 
(Department  of  the  Anny,  2011).  The  Army  is  also  provided  Operating  and  Maintenance 
funding  (OMA)  to  support  sustainment  and  preparation  for  combat  operations.  Poor- 
quality  parts  represent  an  additional  draw  on  both  AWCF  and  OMA  funds. 

In  calendar  year  2010,  the  Army  issued  more  than  $8.2  billion  in  materiel  from 
supply  support  activities  (SSAs)  to  support  organizational  and  intermediate  level  repair 
and  maintenance  of  equipment.  In  addition  to  organization  and  intermediate  level 
maintenance  and  repair,  gross  sales  from  depot  operations  exceed  $6.1  billion  in  2010 
(Department  of  the  Anny,  2011). 

Realizing  the  potential  for  cost  savings  through  improved  operational  efficiency,  the 
Anny  has  initiated  a  variety  of  programs,  including  Lean  Six  Sigma,  which  have  been 
credited  for  hard  savings,  cost  avoidance,  and  increased  capacity  valued  at  $300  million 
per  year  since  FY  2007  (Department  of  the  Anny,  2011).  While  these  initiatives  yielded 
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results,  there  is  still  more  that  can  be  done  to  identify  and  manage  the  sources  of  quality 
deficiencies. 

As  discussed  in  Chapter  2,  the  current  methods  used  to  report  quality  issues  are  not 
automated  and  are,  instead,  highly  dependent  upon  SMEs.  It  is  preferable  that  SME  time 
be  spent  fixing  rather  than  identify ing/detecting  quality  problems.  SMEs  are  essential  to 
identifying  root  causes  and  solutions.  However,  their  time  is  limited,  and  a  process  that 
helps  to  identify  and  prioritize  SME  efforts  would  enhance  their  effectiveness  as  well  as 
improve  efficiency. 

Our  approach  focuses  on  identifying  parts  that  are  failing10  more  frequently  than  in 
the  past,  or  are  failing  more  often  than  similar  parts.  By  identifying  parts  with  unusual 
failure  rates  and  estimating  the  costs  associated  with  the  increased  failures,  we  begin  to 
prioritize  corrective  action  activities.  In  the  remainder  of  this  and  the  following  chapter, 
we  describe  a  method  that  could  be  used  by  the  Army  to  identify  and  prioritize  part 
quality  problems,  which  will  in  turn  enhance  efforts  that  reduce  the  cost  of  quality 
associated  with  poorly  performing  parts.  This  methodology  can  be  efficiently  applied  to  a 
large  number  of  parts  to  create  a  list  of  cost  drivers. 

What  Do  We  Mean  by  the  “Cost  of  Poor  Quality”? 

There  are  many  factors  that  contribute  to  the  cost  of  poor  quality.  PQDRs  are 
primarily  used  to  identify  manufacturing  defects.  However,  the  underlying  cause  of  a 
defect  is  often  complex.  Consider  these  examples: 

•  A  seal  used  on  a  transmission  assembly  was  manufactured  by  a  new  supplier  who 
interpreted  the  technical  drawings  incorrectly,  producing  a  rounded  seal  rather 
than  a  flat  one,  thus  generating  a  PQDR.  However,  the  true  cause  of  the  defect 
was  not  manufacturing,  but  a  technical  drawing  that  did  not  specify  the  shape  of 
the  seal. 

•  A  part  is  sent  for  repair  from  an  organizational  unit  to  an  intermediate  repair 
facility,  where  no  fault  is  found.  This  happens  several  times,  until  a  field  team 
from  the  intermediate  facility  discovers  that  the  organizational  level  maintainers 
did  not  have  the  proper  software  loaded  on  their  test  equipment,  therefore 
generating  false  failures. 

•  An  engine  exits  the  overhaul  line  and  enters  final  test  and  is  tested  multiple  times 
on  a  test  stand.  Each  time  the  repair  technician  removes  the  engine  and  initiates 
repairs  until  the  engine  passes  inspection.  The  “rework”  of  the  engine  ties  up  test 


10  We  assume  that  a  part  is  removed  from  a  weapon  system  because  it  is  no  longer  functioning  as  required. 
A  part  failure  could  be  the  result  of  normal  wear,  improper  maintenance,  or  manufacturing  defects,  or  in 
some  cases  the  part  may  not  have  had  a  fault  but  was  replaced  due  to  incorrect  diagnosis. 
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equipment,  expends  additional  labor  and  material,  and  increases  the  repair  cycle 
time,  yet  this  data  is  not  captured. 

•  A  part  that  is  supposed  to  last  1500  hours  is  failing  at  800  hours,  increasing  the 
inventory  requirement  and  the  maintenance  burden. 

These  are  only  some  examples  of  many — often  hidden — costs  due  to  poor  quality 
documentation,  equipment,  training,  or  parts.  The  cumulative  effect  of  these  costs  not 
only  deteriorates  mission  readiness,  but  can  undermine  confidence  in  and  morale  of  the 
operators  and  maintainers. 

One  of  the  most  commonly  used  models  for  categorizing  cost  of  quality  is  the 
Prevention,  Appraisal,  and  Failure  (P-A-F)  model.  Table  3.1  applies  the  P-A-F  model  to 
some  of  the  processes  identified  in  the  course  of  our  interviews  with  AMC  headquarters, 
field,  and  depot  maintenance  personnel.  Our  definition  of  quality  encompasses  all  of 
these  elements.  Flowever,  it  is  neither  possible,  nor  necessary,  to  capture  all  of  these 
costs.  As  described  in  the  CRC  case  study,  what  is  needed  is  a  starting  point  that  can 
provide  a  measure  of  relative  magnitude.  SMEs  can  then  use  this  information  to  conduct 
root  cause  analysis  and  measure  the  results  of  corrective  actions. 

Table  3.1.  Army  Prevention,  Appraisal,  and  Failure  Cost 


Prevention 

Appraisal 

Failure  Costs 

Phase  inspection 

Supplier  visits 

PQDR  system 

Supplier  visits 

Test  equipment 

Increased  inventory 

Quality  system  administrative 
expenses 

DCMA  receiving  inspection 

Maintenance  actions 

Time  limited  service  life 

Qualification  of  suppliers 

Non-mission-capable  equipment 

Tech  data  maintenance 

Laboratory  support 

Supplier  corrective  action 

Manufacturing  quality  reviews 

Rework 

Design  quality  reviews 

Repair 

Scrap 

Cost  Equations 

We  estimate  the  costs  of  quality  associated  with  increased  inventory  and  maintenance 
actions.  The  other  costs  listed  above,  such  as  rework,  scrap,  PQDR  system,  supplier 
visits,  Defense  Contract  Management  Agency  (DCMA)  inspections,  etc.,  are  not 
considered,  but  could  be  added  if  data  becomes  available. 

Chapter  4  describes  our  method  for  identifying  part  quality  problems  and  the  cost 
equations  used  to  prioritize  parts  that  should  be  considered  for  root  cause  analysis. 
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4.  Methods  for  Identifying  Parts  and  Estimating 
the  Cost  of  Quality 


In  this  chapter  we  summarize  our  approach  to  detecting  potential  part  quality 
problems  and  quantify  the  incremental  cost  incurred  due  to  increased  maintenance 
actions  and  inventory  required  to  meet  readiness  goals. 

Three  Types  of  Poor  Part  Quality  Usage  Patterns  Defined 

We  define  three  types  of  part  use  patterns  that  may  indicate  there  is  a  problem  with 
the  quality  of  a  part  or  an  underlying  (and  perhaps  hidden)  process  deficiency. 

Patterns  of  Type  A  exhibit  a  temporary  increase  in  part  usage.1 1  Figure  4. 1  illustrates 
this  behavior  with  a  temporary  increase  in  part  usage  (y-axis)  for  a  brief  period  of  time 
(x-axis). 


Figure  4.1.  Type  A  Poor  Part  Quality  Behavior 


After  the  temporary  increase,  the  usage  rate  returns  to  its  previous  level.  This  type  of 
behavior  might  occur  if  a  supplier  produces  a  bad  batch  and  subsequently  corrects  the 
quality  problem.  It  might  also  occur  if  the  fleet  is  temporarily  exposed  to  adverse 
conditions,  such  as  a  sandstorm  or  extreme  heat,  both  of  which  subside  with  time, 
returning  the  system  to  its  previous  state. 

Patterns  of  Type  B  exhibit  a  noticeable  shift  in  part  use  that  is  sustained  over  time 
(see  Figure  4.2).  For  example,  a  change  in  a  maintenance  practice  may  result  in  an 
increased  rate  of  removals,  generating  an  increase  in  the  number  of  parts  used. 

1 1  In  this  subsection  we  use  the  term  “usage”  to  capture  the  replacement  rate  at  which  parts  are  being 
“consumed”  within  the  fleet. 
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Figure  4.2.  Type  B  Poor  Part  Quality  Behavior 


The  purpose  of  our  methodology  is  to  detect  the  onset  of  a  Type  A  or  Type  B  part 
quality  issue.  This  early  identification  of  an  emerging  part  quality  issue  does  not  allow  us 
to  determine  whether  the  issue  is  Type  A  or  B.  Only  monitoring  the  issue  after  its  first 
detection  will  reveal  its  type. 

Type  C  compares  the  usage  rates  of  parts  with  similar  form,  fit,  and  function.  In  this 
case,  one  or  more  parts  grouped  together  exhibit  similar  usage  patterns,  while  another 
part  (or  group  of  parts)  exhibits  a  very  different  usage  pattern  (see  Figure  4.3).  A  Type  C 
part  quality  issue  may  or  may  not  be  experiencing  a  temporary  or  sustained  increase  in 
usage,  but  its  overall  usage  rate  is  much  greater  than  similar  parts  across  time. 

Figure  4.3.  Type  C  Poor  Part  Quality  Behavior 


Methods 

To  assess  the  cost  of  poor  part  quality,  first,  methods  are  required  to  automatically 
identify  Type  A,  B  or  C  part  quality  issues  from  among  the  thousands  of  parts  found  on  a 
weapon  system.  Secondly,  these  methods  need  to  provide  quantitative  measures  useful 
for  calculating  the  many  cost  factors  associated  with  the  identified  quality  issue. 
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There  are  several  time-tested  approaches  to  monitor  and  identify  product  quality 
defects.  In  manufacturing,  control  charts  are  used  to  monitor  process  quality.  For 
example,  p-charts  track  the  percent  of  defective  parts  over  time  statistically  computed 
with  seven  rules  for  detecting  special  causes  of  variation.  These  rules  warn  operators  of 
out-of-control  patterns  in  the  number  of  defects.  These  charts  are  used  to  identify 
problems,  which  then  need  to  be  investigated  further  to  identify  and  correct  root  causes. 
Similarly,  in  the  design  and  reliability  engineering  fields,  tools  such  as  Pareto  and 
Weibull  analysis  allow  practitioners  to  monitor  system  performance  over  time  and  assess 
improvements  or  degradation  in  time  between  failures  or  between  repairs. 

In  the  case  of  control  charts,  data  is  usually  readily  available  both  in  terms  of  number 
of  units  produced  and  number  of  defects  or  defective12  items  found.  This  is  not  the  case 
for  the  Army’s  part  usage  data.  Reliability  models,  such  as  Weibull,  are  focused  on 
analysis  of  single  components  on  a  case-by-case  basis.  This  approach  of  assessing  part 
reliability  becomes  very  resource  intensive  when  an  entire  weapon  system  is  of  concern. 
The  result  is  a  limited  reliability  analysis  using  only  simple  measures  like  Mean  Time 
Between  Failure  (MTBF)  that  are  unable  to  decipher  poor  part  quality  behaviors  in  a 
timely  fashion  (e.g.,  Type  A  and  B).  Processing  the  available  data  as  well  as 
automatically  monitoring  and  detecting  unusual  patterns  of  performance  are  predominant 
challenges  that  are  preventing  more  thorough  and  in-depth  analysis  of  all  parts  used  on  a 
weapon  system. 

In  the  following  subsections  we  provide  an  overview  of  the  methods  selected  to 
identify  Type  A,  B,  and  C  part  quality  issues  and  to  estimate  costs.  For  Type  A  and  B 
part  quality  issues,  we  implement  the  Crow-AMSAA  model  (Crow,  1974) 13  along  with  a 
breakpoint  algorithm  that  directs  attention  to  current  part  perfonnance.  For  Type  C 
quality  issues,  we  develop  a  unique  methodology  using  hierarchical  clustering  and  other 
statistical  techniques.  The  last  subsection  introduces  the  cost  model  employed  to  quantify 
the  cost  of  poor  part  quality. 

Method  for  Type  A  and  B  Part  Quality  Issues 

For  Type  A  and  Type  B  quality  issues,  we  want  to  detect  when  the  usage  rate  of  a 
part  is  no  longer  stable  and  begins  to  increase — i.e.,  the  onset  of  an  emerging  part  quality 
issue.  The  “beta”  value14  from  the  Crow-AMSAA  model  quantifies  this  increase. 


12 

“  Note  that  one  defective  part  can  have  multiple  defects  or  modes  of  failure. 

13 

See  Appendix  A  for  a  description  and  references  for  the  Crow-AMSAA  method. 

14 

The  log-log  relationship  between  the  expected  number  of  failures  as  a  function  of  time  and  the 
parameters  A  and  is  expressed  in  by  linear  equation  log  EfAfi)]  =  log  X  +  /?  log  t.  The  parameters  X  and  /? 
can  be  estimated  by  using  several  linear  regression  techniques  including  least  squares,  maximum  likelihood 
estimators  (MLE),  or  generalized  linear  models  (GLM).  See  Appendix  A  for  details. 
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A  beta  value  of  one  indicates  that  part  usage  is  not  changing  over  time;  a  beta  value 
of  less  than  one  indicates  an  increase  in  time  between  failure15  (improved  reliability);  and 
a  beta  value  greater  than  one  indicates  a  decrease  in  time  between  failure  (degrading 
reliability). 

The  Crow-AMSAA  method  requires  cumulative  fleet  time  and  failure  data  to  make 
the  beta  assessment.  Figure  4.4  presents  a  notional  example  of  the  Crow-AMSAA 
method  used  to  identify  a  Type  A  or  Type  B  part  quality  issue  for  a  valve  used  on  Anny 
helicopters. 


Figure  4.4.  Example  of  Crow-ASMAA  Plot 


Every  month,  each  helicopter  in  the  fleet  flies  10  hours,  giving  a  total  of  100  flight 
hours  per  month  for  the  fleet.  The  x-axis  tracks  the  cumulative  fleet  flying  hours,  and  the 
y-axis  tracks  the  cumulative  valve  failures  seen  in  the  fleet.  For  the  first  500  flight  hours 
(the  first  five  months),  a  valve  failure  occurs  every  100  flight  hours.  Fitting  the  Crow- 
AMSAA  model  to  these  points  results  in  a  beta  value  of  one,  indicating  that  the  mean 
time  between  fleet  valve  failures  is  time  invariant  (i.e.,  one  should  expect  the  same 
number  of  valve  part  failures  per  fleet  flight  hour  month  after  month16). 


15  Failure  in  this  case  does  not  necessarily  mean  that  the  part  was  defective.  There  are  multiple  reasons  why 
a  part  may  be  removed  from  a  weapon  system,  including  normal  wear,  incorrect  diagnosis  of  a  problem,  or 
taking  advantage  of  other  maintenance  to  replace  a  part  early. 

16  The  part  removals  count  as  one  failure  or  one  defect.  There  could  be  multiple  causes  for  the  removal  for 
each  part,  but  these  are  not  counted  separately. 
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The  steeper  linear  region  in  Figure  4.4  is  indicative  of  an  increase  in  the  number  of 
failures  per  flying  hour.  In  this  region,  an  additional  valve  failure  occurs  in  the  fleet  every 
100  flight  hours.  In  the  first  100  fleet  hours  there  are  2  failures;  the  next  100  fleet  hours 
there  are  3  failures;  the  next  100  fleet  hours  there  are  4  failures;  etc.  The  beta  value  for 
this  linear  region  is  2.3.  When  the  Crow- AMS AA  beta  is  greater  than  one,  it  indicates 
that  the  average  time  between  part  failures  in  the  fleet  is  decreasing.  Something  has 
happened  that  has  caused  the  helicopter  valve  to  fail  more  frequently:  a  design  change  in 
the  valve;  an  emergent  manufacturing  issue;  the  helicopter  is  operating  under  different 
conditions;  or  a  new  mechanic  is  improperly  trained — these  are  all  plausible 
explanations. 

The  Crow-AMSAA  method  contains  many  characteristics  that  make  it  an  ideal 
method  to  identify  Type  A  and  Type  B  part  quality  issues.  First,  it  is  capable  of  handling 
many  types  of  data  deficiencies,  including  missing  data,  multiple  failure  modes,  batch 
problems,  and  few  data  points.  The  Crow-AMSAA  method  is  able  to  identify  an 
emerging  quality  issue  much  quicker  than  other  statistical  techniques  like  a  moving 
average  (Abemethy,  2004). 

For  the  UFI-60M  case  study,  cumulative  fleet  time  is  obtained  from  the  Logistics 
Information  Warehouse  (LIW)  database.  This  database  tracks  Army  aircraft  flight  time 
and  landings  on  a  monthly  basis.  Order  data  is  retrieved  from  the  Corps/Theater 
Automatic  Data  Processing  Service  Center  (Army)  (CTASC)  database.  CTASC  data 
represent  retail  orders  that  are  made  up  of  orders  from  the  units  that  are  operating  the 
aircraft. 

We  employ  two  schemes  to  limit  the  number  of  false  alarms,  i.e.,  estimating  a  beta 
greater  than  one  when  the  true  beta  is  less  than  or  equal  to  one:  (1)  we  use  a  second  data 
source,  the  Equipment  Downtime  Analyzer  (EDA),  which  captures  the  parts  ordered  in 
response  to  a  failure  that  caused  the  aircraft  to  be  down  overnight,  and  (2)  we  estimate  a 
95  percent  confidence  interval17  around  our  estimate  of  beta  and  use  the  lower  bound  as 
the  indicator  value. 

A  Type  A  or  Type  B  quality  issue  is  identified  when  the  95  percent  lower  confidence 
bound  on  the  estimate  of  the  beta  value  is  greater  than  one  for  both  EDA  and  CTASC 
data.  Figure  4.5  illustrates  a  part  with  a  potential  Type  A  or  Type  B  quality  issue. 


17  A  confidence  interval  around  a  parameter  estimate  establishes  the  range  of  values  that  could  be  observed 
with  repeated  sampling  and  estimation  of  the  parameter.  For  example,  a  95  percent  confidence  bound  of 
1.74  and  1.82  indicates  that  if  100  independent  samples  from  a  population  were  drawn  and  beta  values 
estimated  for  each,  95  out  of  100  of  those  estimates  should  fall  between  1.74  and  1.82.  By  taking  the  lower 
bound  of  the  confidence  interval  as  our  indicator  value,  we  increase  our  confidence  that  the  true  value  of 
beta  is  greater  than  or  equal  to  the  lower  bound. 
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Figure  4.5.  Crow-ASMAA  Plot  for  Component  Found  on  UH-60M  Helicopter 


Crow-AMSAA  Plot  for  ACTUATOR, ELECTRO-ME  (NUN  15579613) 
(SIKORSKY  AIRCRAFT  CORPORATION  (100%),  MikeJJnique,  $19137) 
(CTASC,  EDA,  &  LIW  data  200805  to  201012) 


Figure  4.5  is  a  log-log  plot  of  the  cumulative  flight  hours  (x-axis)  and  the  cumulative 
number  of  demands  (y-axis).  The  triangles  represent  the  plot  of  cumulative  flight  and 
demand  data  using  the  CTASC  data,  while  the  hollow  circles  do  the  same  using  the  EDA 
data.  As  mentioned  earlier  and  detailed  in  Appendix  A,  the  beta  values  are  fit  using  the 
most  recent  linear  region.  We  developed  an  algorithm  that  automatically  works  back 
from  the  most  recent  observation,  adding  previous  observations  until  a  break  is  detected 
in  the  linear  region.  In  this  example,  the  algorithm  has  captured  the  last  seven  CTASC 
data  points  and  last  four  EDA  data  points  for  the  beta  calculation,  the  lines  in  this  figure 
represent  regression  lines,  and  the  slope  of  the  regression  line  has  a  lower  95  percent 
confidence  limit  of  1.74  for  the  CTASC  data  and  1.962  for  the  EDA  data.  In  addition  to 
the  EDA  and  CTASC  data,  the  PQDR  data  are  plotted.  The  box  in  the  upper  left  shows 
that  the  95  percent  confidence  interval  on  the  Crow-AMSAA  betas  for  both  the  CTASC 
and  EDA  data  are  above  one,  flagging  this  part  as  a  potential  problem.  The  solid  circles 
plot  the  cumulative  PQDRs;  this  data  when  it  exists  provides  additional  information  to 
support  our  findings.18 


1 8 

1  As  noted  in  Chapter  2,  PQDRs  are  not  filed  for  every  quality  issue.  Therefore,  we  cannot  use  PQDR  data 
to  validate  our  methods,  but  it  does  serve  as  supporting  information. 
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Type  C  Part  Quality  Identification 

Type  C  part  quality  issues  are  concerned  with  parts  that  exhibit  a  higher  usage  rate 
than  similar  parts.  We  follow  a  two-step  approach  to  identify  these  parts.  First,  we  cluster 
like  parts  together.  We  use  a  hierarchical  clustering  method  to  group  similar  parts  within 
a  Federal  Stock  Class  (FSC)  based  on  part  features  including  part  nomenclature  (general 
description  of  the  part)  and  price,  weight,  and  dimensions  (height,  width,  and  length).19 
Second,  we  compare  the  usage,  normalized  by  flight  hours,  of  each  part  in  the  cluster  and 
identify  if  any  outliers  exist.  We  calculate  the  part  usage  in  this  second  step  as  the 
number  of  orders  per  month  seen  in  the  CTASC  data  divided  by  the  cumulative  flight 
hours  logged  on  that  part. 

In  Figure  4.6  we  show  an  example  of  a  part  that  has  a  much  higher  usage  than  similar 
parts  that  were  grouped  within  the  same  cluster.  The  figure  shows  one  a  switch  guard  that 
has  a  much  higher  usage  rate  compared  to  other  switch  guards  with  similar  price,  weight, 
and  dimensions.  In  the  second  step  of  our  clustering  method  we  use  a  statistical  test  to 
identify  outliers,  allowing  for  full  automation  of  this  method.  See  Appendix  B  for  further 
details. 


Figure  4.6.  Example  of  Clustering  Method  Results 
Cluster  Type:  All  Hierarchical,  Standardized  CTASC,  FSC  5930 

FSC  Nomenclature:  Switches 
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See  Appendix  B  for  more  details  on  the  clustering  methodology. 
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Estimating  the  Cost  of  Poor  Part  Quality 

As  described  in  Chapter  3,  we  will  focus  our  efforts  on  estimating  how  increased  part 
use  due  to  potential  quality  problems  will  increase  the  costs  of  maintenance  and 
inventory  needed  to  meet  readiness  goals.  We  start  with  the  list  of  parts  identified  as 
potential  Type  A,  B,  or  C  quality  issues  and  estimate  the  increased  costs  if  current  failure 
rates  are  not  reduced. 

Cost  of  Increased  Maintenance 

Our  calculation  of  maintenance  costs  is  limited  to  the  maintenance  activity  at  the 
organizational  level.  In  most  cases,  parts  are  not  repaired  at  the  organizational  level. 
Rather,  parts  are  replaced  and  sent  to  the  authorized  maintenance  activity  for  repair. 

When  a  part  is  removed  from  an  end  item  at  the  organizational  level,  the  replacement 
part  is  purchased  from  the  Army  Working  Capital  Fund,  typically  from  an  OMA  account. 
For  reparable  parts,  the  exchange  price  reflects  the  effective  cost  of  the  item  (direct  and 
indirect  labor  to  repair  the  part  and  any  surcharge).  For  consumable  parts,  the  price  is 
provided  by  the  latest  unit  price  in  the  Fed  Log,20  which  includes  an  administrative 
surcharge.  We  did  not  include  labor  charges  at  the  organizational  level  because  actual 
repair  hours  are  not  recorded,  and  we  assume  the  cost  of  direct  labor  is  a  sunk  cost  at  this 
level.  Flowever,  the  model  could  easily  be  expanded  to  include  an  estimate  of  this  cost 
using  standard  hours  and  the  frequency  of  exchange,21  which  would  allow  us  to  capture 
the  total  or  true  cost  associated  with  the  failure  of  parts  on  an  end  item. 

To  estimate  incremental  cost  due  to  poor  quality,  we  need  a  baseline  of  the  number  of 
replacements  ( Qq )  that  would  take  place  at  the  organizational  level  if  the  part  was 
performing  within  acceptable  quality  standards  (beta  =1),  and  an  estimate  of  the 
replacements  (Q i)  if  the  part  was  not  perfonning  to  standard  (beta  >  1).  For  a  Type  A  and 
B  quality  issue,  Qq  is  the  expected  number  of  failures  if  the  emerging  quality  issue  never 
occurred  (the  beta  value  never  increased  above  one).  Q\  is  the  number  of  failures  we 
expect  to  see  under  the  current  quality  issue  (the  forecast  of  failures  using  the  estimated 
beta  value  that  is  greater  than  one).  For  the  Type  C  quality  issue,  Qq  is  the  expected 
failures  of  the  similar  parts  that  fell  within  the  cluster,  whereas  Q\  is  the  expected  failures 
of  the  part  under  investigation.  We  develop  a  forecast  of  Qq  and  Q\  over  the  next  six 
months  to  generate  a  six-month  forecast  of  costs.  Thus,  equation  (1)  is  the  cost  due  to 
increased  maintenance: 


20 

Federal  Logistics  Information  Service  (Defense  Logistics  Agency,  2011). 

21 

Many  end  item  repairs  may  involve  several  parts,  so  the  standard  labor  hours  would  have  to  be  allocated 
across  the  parts. 
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Cost  due  to  increased  maintenance  is: 


(Q i  -  go)  *  price 


(1) 


Cost  of  Increased  Inventory 

We  assume  the  Army  will  adjust  inventory  levels  to  maintain  required  readiness  and 
estimate  inventory  levels  under  the  two  scenarios  described  above.  We  use  the  Customer 
Service  Level  (CSL)  fonnula  (Vollmann,  2005)  to  solve  for  the  reorder  point  (ROP), 
assuming  an  85  percent  fill  rate22  target,  and  using  demands  from  all  SSAs  off  the  shelf  to 
the  organizational-level  customer.  We  compute  a  ROPo  for  the  baseline  quality  scenario 
and  a  ROPi  under  the  deteriorated  quality  scenario. 


CSL.  =  I00-(100/E0a)5>(4)(4 -rop.) ,  (2) 

d= i 


where 


do  =  the  forecast  of  demand  over  the  next  six  months  if  the  quality  problem  is  fixed 

d\  =  the  forecast  of  demand  over  the  next  six  months  if  the  quality  problem  persists 

P(d\)  =  probability  of  demand  if  reduced  quality 

P{do)  =  probability  of  baseline,  or  problem  is  fixed 

rop  i  =  reorder  point  if  problem  persists 

ropo  =  baseline  or  reorder  point  if  problem  is  fixed 

EOQ\  =  economic  order  quantity  if  problem  persists 

EOQo=  baseline  or  economic  order  quantity  problem  fixed 


Incremental  inventory  costs  due  to  poor  quality  are: 

(Average  IP  problem  persists  -  Average  IP  problem  fixed)  *  Price  , 


where 


IP  =  Inventory  position  =  rop  +  A  EOQ. 


22 

“  The  fill  rate  parameter  can  be  varied  as  desired,  and  an  85  percent  fill  rate  target  is  consistent  with  the 
Army’s  retail  inventory  performance  goals. 
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5.  UH-60M  Case  Study  Results 


The  UH-60M  is  the  newest  variant  of  the  Black  Hawk  utility  helicopter  family,  with 
inductions  of  new  UH-60M  models  beginning  in  2007.  This  model  has  many  upgraded 
components  and  unique  systems  that  are  not  found  on  the  older  UH-60L  and  UH-60A 
models.  In  particular,  the  UH-60M  model  has  an  “all  glass”  cockpit  that  contains  all 
digital  electronics  and  displays  that  are  unlike  the  analog  instruments  found  on  the  UH- 
60A  and  UH-60L.  As  such,  the  operational  reliability  performance  of  these  new 
components  is  not  well  known.  Since  this  model  is  still  in  manufacturing,  issues 
discovered  in  the  field  can  be  corrected  during  production,  which  is  generally  less  costly 
than  correcting  problems  in  the  field.  In  addition,  the  UH-60M  model  has  been  heavily 
utilized  in  current  combat  zones.  For  these  reasons,  the  Black  Hawk  Program  Office  has 
been  closely  tracking  the  reliability  performance  of  the  UH-60M  model  (RAND,  2011). 

The  Black  Hawk  Program  Office  Reliability  and  Maintainability  (RAM)  group  is 
primarily  responsible  for  tracking  the  reliability  of  the  UH-60M  model.  This  office  works 
in  conjunction  with  engineers  from  the  Aviation  and  Missile  Research  Development  and 
Engineering  Center  (AMRDEC),  who  manually  “score”  logbook2’  data  to  identify 
systems  that  are  causing  abnormal  aircraft  downtime.  The  Black  Hawk  RAM  office  also 
incorporates  information  from  the  field  using  PQDRs  to  identify  parts  that  need 
immediate  attention  or  that  should  be  considered  for  future  redesign.  It  is  important  to 
note  that  the  primary  focus  of  this  office  is  on  the  manufacturing  and  design  reliability  of 
the  UH-60  and  its  component  systems.  Reviews  of  the  AMRDEC  data  and  other  systems 
intelligence  are  used  biannually  to  identify  and  execute  fixes  for  systems  that  need 
corrective  action  in  the  Black  Hawk  UH-60M  fleet.  This  attention  to  the  UH-60M  model 
has  been  instrumental  in  providing  information  to  gauge  the  application  and  validity  of 
our  methodologies. 

Type  A  or  B  Poor  Quality  Parts  on  the  UH-60M 

For  this  part  of  the  case  study,  we  analyze  a  total  of  5,910  parts  that  are  found  on  the 
UH-60M.  Though  this  helicopter  has  many  more  parts,  we  limit  our  analysis  to  parts  that 
caused  the  UH-60M  to  be  out  of  service  overnight  as  recorded  in  the  EDA  data.  We 
examined  CTASC  and  EDA  data  for  the  April  2008  and  December  2010  timeframe. 
Using  the  Crow-AMSAA  method,  a  total  of  34  parts  were  identified  as  having  a  Type  A 

23 

Each  aircraft  has  an  electronic  logbook  that  records  the  flying  and  maintenance  events  on  the  aircraft. 
The  data  format  is  not  conducive  to  automated  analysis,  but  requires  manual  interpretation  from  subject 
manner  experts. 


25 


or  Type  B24  quality  issue.  We  prioritized  these  34  parts  using  our  cost  calculation  (repair 
plus  inventory  costs)  and  present  the  top  10  in  Table  5.1. 

The  parts  are  prioritized  by  quality  costs  in  the  column  labeled  “Total  CoQ,”  and  the 
actual  part  price  is  identified  in  the  third  column.  The  noteworthy  observation  from  the 
comparison  of  the  total  CoQ  and  the  price  of  the  part  is  that  a  high-priced  part  does  not 
necessarily  correspond  with  a  high  cost  due  to  poor  quality  (e.g.,  the  control  generator 
with  unit  price  -$5,000  has  higher  cost  implications  than  the  rotary  blade  with  unit  price 
-$200,000).  There  are  two  reasons  for  this:  (1)  the  CTASC  beta25  is  -1.6  for  the  control 
generator,  much  greater  than  the  beta  of  -1 .08  for  the  rotary  blade;  and  (2)  the  volume  of 
orders  in  calendar  year  20  1  026  is  much  greater  for  the  generator.  In  fact,  the  rotary  blade 
is  driven  onto  the  list  by  its  price.  The  beta  value  is  only  slightly  higher  than  one,  which 
is  not  a  significant  indicator  of  a  quality  issue.  However,  due  to  the  price  of  this  item, 
further  investigation  is  warranted.  Even  a  small  decrease  in  beta  through  a  change  in 
design  or  maintenance  could  produce  worthwhile  cost  savings. 

The  last  column  of  Table  5.1,  2010  EDA  N,  indicates  how  many  recorded  downtime 
events  were  associated  with  the  particular  part  in  the  EDA  data.  This  column  is  used  to 
gauge  the  readiness  impact  caused  by  the  part  quality  issue.  For  example,  the  rotary  blade 
was  associated  with  12  overnight  downtime  events,  whereas  the  electro-mechanical 
actuator  unique  to  the  UH-60M  model  had  nearly  double  (23)  such  events  in  a  year’s 
time. 

The  “PQDR”  column  of  Table  5. 1  is  binary  and  identifies  whether  the  part  has  had  at 
least  one  PQDR  submitted  in  the  time  period  examined  (April  2008  to  December  2010). 
For  most  high-priced  items,  at  least  one  PQDR  was  submitted.  However,  this  is  not 
always  the  case  for  lower-priced  items  (under  $100).  As  discussed  in  Chapter  2,  there  are 
multiple  reasons  why  a  part  failure  might  not  generate  a  PQDR.  For  example,  NUN 
015 118345, sensor  unit  laser,  priced  at  $36,000,  has  abeta  value  of-1.3  and  has 
experienced  nine  EDA  deadlining  events  in  2010  (2010_EDA_N)  in  calendar  year  2010, 
yet  no  PQDRs  were  filed. 
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The  only  difference  between  a  Type  A  and  Type  B  error  is  the  behavior  after  the  uptick  in  poor  part 
quality  behavior.  Only  time  will  reveal  if  these  part  quality  issues  will  remain  (Type  B)  or  self-correct 
themselves  by  returning  to  the  same  usage  level  (Type  A). 

“5  Refer  to  the  “CTASC  Beta”  column  in  Table  5.1. 

“6  Refer  to  the  “CTASC_2010_Q”  column  in  Table  5.1. 
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Table  5.1.  UH-60M  Parts  with  Type  A  and  B  Indicators  as  of  December  2010 


niin 

Nomen 

Price 

PQDR 

AC 

CTASC 

Beta 

Percent 

Ordered 

from 

Vendor 

Repair  Cost 

Inventory 

Cost 

Total  CoQ  2010 

CTASCQ 

Z 

< 

Q 

LU 

1 

o 

o 

CN 

015046723 

CONTROL, GENERATOR 

$4,852.00 

yes 

Mult_AC 

1.625 

93% 

$2,209,120.25 

$526,442.00 

$3,126,321.14 

390 

24 

015461148 

BLADE, ROTARY  WING 

$199,336.00 

yes 

Mike_Unique 

1.081 

80% 

$853,106.87 

$398,672.00 

$1,254,699.93 

12 

4 

015004770 

RECEIVER-TRANSMITTE 

$46,217.00 

yes 

Mult_AC 

1.330 

84% 

$562,335.67 

$600,821.00 

$1,206,803.08 

77 

24 

011625035 

SERVOVALVE, HYDRAULI 

$10,090.00 

yes 

Mult_AC 

1.294 

33% 

$746,781.53 

$292,610.00 

$1,039,391.53 

282 

16 

012988467 

ACCUMULATOR, PNEUMAT 

$9,629.00 

yes 

Mult_AC 

1.408 

NA 

$615,275.35 

$365,902.00 

$981,177.35 

279 

7 

015579613 

ACTUATOR, ELECTRO-ME 

$19,137.00 

yes 

Mike_Unique 

1.777 

100% 

$381,653.48 

$172,233.00 

$577,397.13 

23 

3 

010892850 

BRAKE, MULTIPLE  DISK 

$1,878.41 

yes 

Mult_AC 

1.138 

97% 

$393,413.25 

$82,650.04 

$476,063.29 

907 

70 

015118345 

SENSOR  UNIT, LASER  D 

$36,875.00 

no 

Mike_Unique 

1.319 

69% 

$42,750.60 

$221,250.00 

$284,212.68 

41 

9 

015588744 

COMPUTER, FLIGHT  CON 

$70,160.00 

yes 

Mike_Unique 

1.171 

NA 

$48,056.75 

$210,480.00 

$269,724.52 

25 

4 

013917116 

PANEL, FAULT-FUNCTIO 

$4,293.66 

yes 

Mult_AC 

1.365 

NA 

$172,341.64 

$49,377.09 

$257,602.67 

67 

13 

015541623 

SKIN, AIRCRAFT 

$343.00 

yes 

Mike_Unique 

1.569 

100% 

$50,095.40 

$12,176.50 

$171,080.31 

138 

5 

012612044 

ACCTUATOR, ELECTRO-ME 

$7,804.00 

yes 

Mult_AC 

1.013 

12% 

$120,006.43 

$31,216.00 

$151,222.43 

942 

53 

014965565 

STRAP, WEBBING 

$1.50 

no 

Mult_AC 

1.049 

100% 

$329.17 

$66.00 

$135,271.19 

2881 

11 

013363497 

CABLE  ASSEMBLY, SPEC 

$616.54 

yes 

Mult_AC 

1.092 

100% 

$110,115.66 

$22,503.71 

$132,619.37 

1140 

37 

011396338 

SWITCH, PRESSURE 

$318.75 

yes 

Mult_AC 

1.575 

100% 

$99,201.41 

$23,906.25 

$123,107.66 

267 

16 

011373398 

TIRE, PNEUMATIC, AIRC 

$607.95 

yes 

Mult_AC 

1.023 

100% 

$4,172.19 

$7,903.35 

$49,075.54 

1796 

53 

010986005 

PACKING  WITH  RETAIN 

$4.19 

yes 

Mult_AC 

1.404 

100% 

$39,382.44 

$8,136.98 

$47,519.42 

13223 

108 

011269456 

INNER  TUBE, PNEUMATI 

$53.75 

no 

Mult_AC 

1.244 

100% 

$22,931.23 

$4,676.25 

$27,607.48 

996 

9 

008892495 

RIVETXSOLID 

$15.35 

no 

Mult_AC 

3.745 

100% 

$16,605.58 

$5,587.40 

$22,192.98 

70 

38 

008033044 

WIRE, NONELECTRICAL 

$76.22 

no 

Mult_AC 

1.219 

100% 

$14,754.09 

$3,201.24 

$17,955.33 

804 

943 

014982223 

BATTERY, STORAGE 

$1,889.56 

yes 

Mult_AC 

1.045 

100% 

$10,760.53 

$1,889.56 

$17,377.52 

85 

13 

011056582 

LIGHT, LANDING, AIRCR 

$3,717.00 

yes 

Mult_AC 

1.014 

15% 

$9,644.71 

$3,717.00 

$13,361.71 

233 

21 
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NUN 

Nomen 

Price 

PQDR 

AC 

CTASC 

Beta 

Percent 

Ordered 

from 

Vendor 

Repair  Cost 

Inventory 

Cost 

Total  CoQ  2010 

CTASCQ 

2010EDAN 

005589763 

NUTXCASTELLATEDXHEX 

$9.58 

no 

Mult_AC 

1.281 

100% 

$9,433.56 

$1,963.00 

$11,397.46 

2224 

27 

011080476 

LEAD, ELECTRICAL 

$23.81 

yes 

Mult_AC 

1.116 

100% 

$7,451.35 

$1,500.03 

$8,951.38 

1617 

20 

010144596 

O-RING 

$4.98 

yes 

Mult_AC 

1.134 

100% 

$4,441.23 

$901.38 

$5,342.61 

5103 

61 

011053633 

HANDLE, DOOR 

$14.55 

no 

Mult_AC 

1.117 

100% 

$2,884.13 

$625.65 

$3,509.78 

1319 

10 

01651958 

O-RING 

$0.91 

no 

Mult_AC 

1.358 

100% 

$1,091.82 

$247.07 

$1,338.89 

2188 

70 

011539682 

WEAR  STRIP, CARGO  DO 

$4.69 

no 

Mult_AC 

1.018 

100% 

$875.08 

$171.19 

$1,046.27 

5911 

60 

011247622 

SEALING  COMPOUND 

$11.90 

no 

Mike_Unique 

1.213 

NA 

$686.99 

$160.65 

$847.64 

121 

13 

001451161 

LAM  P,  INCAN  DESENT 

$26.74 

no 

Mult_AC 

1.035 

100% 

$551.82 

$120.33 

$672.15 

486 

24 

001557784 

LAMP.INCANDESENT 

$0.19 

no 

Mult_AC 

1.510 

NA 

$463.50 

$117.42 

$580.92 

3198 

80 

011234601 

SCREW, MACHINE 

$0.16 

no 

Mult_AC 

1.068 

100% 

$161.81 

$36.24 

$198.05 

9631 

75 

000103090 

CLIP, RETAINING, AVIA 

$0.14 

no 

Mult_AC 

1.028 

100% 

$102.92 

$21.35 

$124.27 

18324 

44 

001611017 

O-RING 

$0.13 

no 

Mult_AC 

1.007 

100% 

$2.66 

$0.91 

$3.57 

2113 

14 

28 


An  important  aspect  of  this  methodology  is  to  identify  both  parts  with  poor  quality  as 
well  as  the  associated  suppliers.  In  Table  5.1,  the  “%  Order  from  Vendor”  column 
identifies  the  portion  of  parts  ordered  from  that  supplier  in  the  last  year.  For  example,  12 
percent  of  the  electro-mechanical  actuators  (NUN  012612044)27  were  ordered  from  an 
external  supplier,  while  the  other  88  percent  were  received  as  a  reworked  part  from  the 
Army’s  maintenance  depot.  This  identified  part  quality  problem  may  not  be  an  issue  with 
the  vendor,  but  with  how  the  depot  is  reworking  the  part.  Further  investigation  would  be 
required  to  answer  this  question. 

UH-60M  Parts  with  Type  C  Poor  Part  Quality  Behavior 

In  this  section  of  the  case  study,  we  analyze  a  total  of  3,363  parts  that  are  used  on  the 
UH-60M.  This  analysis  is  somewhat  more  limited,  since  we  examine  only  a  portion  of 
the  FSCs  associated  with  the  5,910  parts  that  we  examined  for  the  Type  A  or  B 
component  of  this  case  study.  Out  of  the  3,363  parts  examined,  83  were  identified  with  a 
Type  C  part  quality  issue.28  Table  5.2  lists  the  top  ten  Type  C  part  quality  issues  by  their 
cost. 

Many  of  the  columns  in  Table  5.2  are  similar  to  those  in  Table  5.1  (price,  PQDR,  % 
from  Vendor,  and  total  cost).  The  additional  columns  are  Mean  Time  Between 
Replacement29  (MTBR)  for  the  cluster,  the  MTBR  for  the  NUN,  and  the  difference 
between  the  two  (MTBR  Delta).  For  example,  the  cluster  containing  the  flight  control 
computer  (NUN  015588744)  has  an  overall  MTBR  of  3,644  hours.  This  means  that  on 
average  there  is  an  expected  part  replacement  every  3,644  flight  hours  for  parts  contained 
in  this  cluster.  The  MTBR  of  the  flight  control  computer,  listed  first  in  Table  5.2,  is  684 
hours.  Flaving  to  replace  a  flight  control  computer  every  684  hours  is  a  significantly 
higher  rate  compared  to  like  parts  that  have  an  MTBR  of  3,644  hours,  if  the  increased 
failure  rate  persists,  the  projected  cost  to  the  Army  is  $3.4  million  in  additional  inventory 
and  maintenance  costs  over  the  next  six  months.  If  it  was  determined  that  the  difference 
in  MTBR  was  due  to  a  design  flaw,  then  a  redesign  of  this  part  could  be  warranted  if  it 
could  increase  the  MTBR  from  684. 
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NUN  or  National  Item  Identification  Number,  is  a  9-digit  numeric  code  which  uniquely  identifies  an 
item  of  supply. 

28 

We  continue  to  refine  the  clustering  algorithm  to  improve  the  initial  clusters  and  reduce  false  positive 
identifications. 
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MTBR  is  similar  to  MTBF  (mean  time  between  failure).  The  difference  is  that  here  we  are  using  order 
data  instead  of  failure  data. 
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Table  5.2.  UH-60M  Parts  with  Type  C  Use  Patterns 
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NA 
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Discussion  of  Select  Parts  Identified  with  a  Type  A,  B,  or  C 
Part  Quality  Issue 

In  Table  5. 1,  we  identify  the  control  generator  as  a  top  quality  driver  with  a  Type  A 
or  B  quality  issue.30  This  part  has  an  acquisition  price  of  just  under  $5,000,  yet  it  is 
identified  to  have  a  greater  cost  impact  than  the  main  rotor  blades,  which  cost  nearly 
$200,000.  The  control  generator  usage,  along  with  its  higher  CTASC  beta  value,  is 
causing  this  part  to  have  a  greater  cost  impact  than  the  rotary  blade.  The  total  retail  orders 
recorded  in  CTASC  for  the  generator  were  390  versus  12  for  the  blade  (see 
2010CTASCQ  column  in  Table  5.1).  In  addition,  the  PQDR  column  in  Table  5.1 
indicates  that  at  least  one  PQDR  has  been  submitted  for  this  part.  In  the  Figure  5.1  below, 
the  cumulative  PQDR  submissions  are  compared  to  the  cumulative  EDA  events. 

Figure  5.1.  PQDR  Submissions  and  EDA  Events  on  Crow-AMSAA  Plot  for  Control 

Generator 


30 

The  Crow-AMSAA  method  identifies  an  emerging  part  quality  issue  at  its  onset.  To  distinguish  between 
a  type  A  or  B  issue  requires  observing  the  beta  value  over  the  next  several  months. 
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For  this  particular  part,  the  frequency  of  PQDR  submissions  and  EDA  events 
simultaneously  increased.  The  same  observation  holds  when  the  CTASC  data  is 
compared  to  the  PQDR  submissions.  This  data  was  presented  to  the  Blackhawk  RAM 
office  who’s  data31  from  AMRDEC  (based  mostly  on  reports  from  operations  in 
Afghanistan)  had  not  indicated  a  problem  with  the  control  generator.  Examining  the 
narratives  in  the  PQDR  reports  submitted  for  this  generator  did  not  provide  additional 
information  except  that  there  may  be  a  problem  with  generators  shorting.  However, 
further  investigation  revealed  that  control  generators  short-out  due  to  a  high  power  wash 
the  helicopter  receives  when  it  is  brought  back  from  theater.  This  insight  was  provided  by 
a  former  crew  chief  who  was  responsible  for  maintenance  of  fielded  Army  helicopters. 

The  second  top  type  A  or  type  B  quality  issue  is  the  UH-60M  main  rotary  blade. 
Figure  5.2  shows  the  Crow- AMS AA  plot  for  this  NUN. 

Figure  5.2.  UH-60  Main  Rotary  Blade  Crow-AMSAA  Plot 


Cumulative  Fleet  Flight  Time  (Hours) 
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The  purpose  of  the  Blackhawk  RAM  office  and  its  collaboration  with  AMRDEC  is  to  eliminate 
reliability  issues  affecting  safety  and  readiness.  Because  of  the  heavy  use  of  the  UH-60M  in  Afghanistan, 
the  RAM  office  has  focused  on  reports  from  the  theater.  Our  approach  is  examining  the  cost  implications  of 
poor  part  quality  across  the  entire  fleet  regardless  of  where  the  helicopters  are  operating. 
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The  Crow-ASMAA  plot  for  the  UH-60M  rotary  blade  indicates  a  slight  increase  in 
order  and  repair  frequency  with  the  beta  95  percent  confidence  intervals  barely  exceeding 
one.  This  slight  increase  is  just  enough  to  make  this  part  second  in  the  Type  A  or  B  part 
quality  issue  list  in  Table  5.1.  To  validate  this  finding,  we  obtained  a  copy  of  the 
logbook32  data  that  is  scored  by  AMRDEC  and  analyzed  it  using  the  Crow-AMSAA 
method.  While  the  data  plotted  in  Figure  5.2  includes  all  aircraft,  the  analysis  using  the 
logbook  data  seems  to  indicate  that  rotor  blade  removals  are  a  much  larger  issue  for 
UH-60M  helicopters  in  Afghanistan,  with  a  beta  value  of  ~2.3  (Figure  5.3). 

Figure  5.3.  Crow-AMSAA  Analysis  of  Logbook  Data  on  the  UH-60M  Main  Rotor  Blade 


The  logbook  data33  plotted  in  Figure  5.3  captures  scheduled  and  unscheduled  rotor 
blade  repair  and  replacement.  There  is  a  distinctive  bend  in  the  plot,  at  around  5,000 
flight  hours,34  that  indicates  an  increasing  frequency  in  maintenance  events  for  the  rotary 

32 

The  logbook  data  requires  manual  scoring  to  make  it  suitable  for  analysis.  Due  to  the  vast  volume  of  data 
and  time  required,  only  10  percent  of  aircraft  logbook  data  is  scored,  most  of  it  for  aircraft  located  in 
Afghanistan. 
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Each  Army  helicopter  maintains  an  electronic  logbook.  Maintainers  will  enter  maintenance  dialog  that 
captures  repairs  done  on  the  helicopter.  These  logs  are  in  free-flow  narrative  text  fields. 

34 

The  total  number  of  fleet  hours  in  this  plot  using  the  logbook  data  is  much  less  than  the  other  plots  using 
the  CTASC  and  EDA  data,  since  only  10  percent  of  the  logbook  data  is  scored,  i.e.,  cleansed  enough  for 
analysis.  Further,  the  logbook  data  was  predominantly  from  aircraft  deployed  in  Afghanistan,  where  the 
environmental  conditions  resulted  in  the  coating  of  the  blades. 
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blade.  By  examining  the  narratives  in  the  logbook,  it  appears  there  are  two  potential 
issues.  First,  approximately  61  percent  of  the  unscheduled  events  require  repair  or 
reapplication  of  a  coating  that  protects  the  blade  from  desert  sand  and  grit.  This  type  of 
repair  does  not  require  a  replacement  of  the  blades,  and  would  not  be  captured  by  the 
EDA  or  CTASC  data.  This  explains  why  the  Crow-AMSAA  plots  using  CTASC  and 
EDA  do  not  indicate  a  significant  issue  (see  Figure  5.2).  Flowever,  the  number  of  repairs 
of  the  coating  are  not  increasing  or  decreasing  through  the  time  period  examined, 
indicating  that  this  is  not  the  issue  causing  the  bend  in  the  curve. 

The  second  issue  revealed  by  the  logbook  narratives  deals  with  rotary  blade 
replacement:  there  are  no  blade  replacements  in  the  first  5,000  flight  hours;  there  are  six 
blade  replacements  between  5,000  and  10,000  flight  hours,  and  half  of  those  are  due  to  an 
inability  to  balance  the  blades.  This  significant  increase  in  blade  replacements  most  likely 
caused  the  bend  in  the  Crow-AMSAA  plot  in  Figure  5.3.  From  the  data  we  could  not 
determine  the  reason  for  problems  balancing  the  blades;  this  would  require  further 
investigation.  One  hypothesis  is  that  repeated  coatings  of  the  blades  along  with  the  harsh 
environmental  conditions  in  Afghanistan  could  be  contributing  to  balancing  problems. 
Since  these  are  replacement  repairs,  this  issue  most  likely  caused  the  slight  increase  in  the 
beta  of  the  fleet  demand  data  for  the  blades — see  Figure  5.2 — although  this  is  less 
pronounced  than  the  logbook  because  it  is  based  on  demands  for  all  helicopters,  not  just 
those  with  coated  blades  in  Afghanistan. 

Figure  5.4  compares  the  performance  (standardized  by  flight  hours)  of  four  rotary¬ 
wing  blades  used  on  the  UH-60  A-M  models.  The  NSNs  labeled  UH-60  (symbols  +,  x) 
are  unique  to  the  UH-60M  model.  A  close  examination  of  the  points  in  Figure  5.4 
indicates  that  the  perfonnance  of  the  coated  rotary  blade  (NUN  015461 148)  appears 
slightly  better  than  that  of  the  noncoated  blades.  However,  this  performance  may  not  hold 
out  in  the  future,  as  the  Crow-AMSAA  analysis  identified  that  the  frequency  of  blade 
replacements  is  slowly  increasing.  The  conclusion,  in  this  case,  is  that  this  part  should 
continue  to  be  monitored  closely. 
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Figure  5.4.  Cluster  Analysis  Comparing  UH-60M  Main  Rotary  Blade 
with  Similar  Rotary-Wing  Blades 
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Another  part  identified  as  a  Type  C  quality  issue  was  an  electro-mechanical  actuator 
found  only  on  the  UH-60M  model  (NUN  015579613).  The  cluster  containing  this  part  is 
shown  in  Figure  5.5. 
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Figure  5.5.  Cluster  Comparison  of  Electro-Mechanical  Actuator 
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Of  the  eight  actuators  in  this  cluster,  two  have  much  higher  usage  compared  to  the 
others.  The  UH-60M  unique  actuator  has  a  MTBR  of  980  hours,  whereas  the  average 
MTBR  of  the  cluster  is  2,488  hours.  When  this  data  was  presented  to  the  Black  Hawk 
RAM  office  and  AMRDEC,  they  were  unaware  of  any  potential  issues  with  this  part,  but 
we  were  infonned  that  the  UH-60M  model,  along  with  the  Lima  and  Alpha  models,  have 
a  similar  actuator’5  that  has  been  approved  to  go  through  a  reliability  improvement 
program.  We  then  talked  to  the  item  manager  for  this  actuator,  who  was  not  aware  of  any 
specific  issues  but  indicated  that  the  demands  for  the  part  have  been  increasing. 

This  case  study  identifies  over  100  parts  that  exhibit  emerging  or  existing  reliability 
issues.  After  we  provided  these  100  parts  to  the  Anny — prioritized  by  their  potential 
incremental  cost  due  to  poor  quality — problems  with  many  of  the  parts  were  verified  by 
SMEs.  In  one  instance  where  SMEs  were  unable  to  provide  confirmatory  evidence, 
further  investigation  revealed  that  the  SMEs  had  limited  visibility  of  potential  problems, 
or  the  part  quality  problem  was  narrowly  defined  to  manufacturing  defects,  thus 
eliminating  potential  process  problems  (e.g.,  the  power  wash  that  was  increasing  control 
generator  failures). 
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Differing  nomenclature  is  the  reason  these  actuators  were  not  clustered  with  the  UH-60M’s.  When  the 
nomenclature  was  adjusted,  all  these  actuators  clustered.  We  are  currently  examining  ways  to  adjust  for 
different  nomenclatures. 
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6.  Observations  and  Recommendations 


This  research  explored  the  feasibility  of  using  readily  available  demand  and  end  item 
maintenance  history  to  identify  potential  issues  with  part  quality  and  to  estimate  their 
associated  cost  impacts.  The  value  in  the  approach  we  have  presented  comes  from  its 
power  to  analyze  the  perfonnance  of  thousands  of  parts,  and  by  assigning  a  cost,  to 
prioritize  a  list  of  parts  with  the  highest  potential  return  on  investment. 

The  methods  presented  can  be  used  to  identify  possible  part  or  process  quality  issues 
by  weapon  system  or  across  weapon  systems.  These  can  then  be  reported  and  corrective 
actions  tracked  by  different  levels  of  AMC  management. 

The  initial  case  study  presented  in  this  report,  continuing  analysis,  and  conversations 
with  subject  matter  experts  all  indicate  that  the  methods  developed  during  this  phase  of 
our  research  demonstrate  potential  for  identifying  problem  parts  and  processes.  The 
proposed  approach  can  be  viewed  as  the  first  step  of  a  multi-step  quality  management 
process,  i.e.,  it  identifies  and  prioritizes  potential  problems;  however,  it  does  not  reveal 
the  root  causes  of  problems.  The  information  provided  should  be  incorporated  with  other 
existing  part  quality  management  data,  such  as  the  PQDRs  and  logbook  reports,  to  select 
quality  improvement  projects  with  the  highest  potential  rate  of  return. 

Various  Organizations  Are  Concerned  with  Part  and  Process  Quality 

In  Chapter  2  we  described  how  the  Army  currently  reports  part  quality  problems 
using  PQDRs  and  subject  matter  experts.  In  addition,  there  are  various  organizations 
responsible  in  one  way  or  another  for  assuring  quality.  These  include: 

•  the  program  offices; 

•  RDECOM  organizations  such  as  Aviation  and  Missile  Research  Development  and 
Engineering  Center  (AMRDEC),  which  provide  engineering  support  to  program 
offices  and  conduct  supplier  audits; 

•  AMC/G-4  Continuous  Process  Improvement  (CPI),  a  recently  established  group 
responsible  for  setting  policy  on  quality  for  AMC; 

•  the  Defense  Contract  Management  Agency  (DCMA),  responsible  for  overseeing 
contract  requirements  and  assuring  that  items  confonn  to  specifications;  and 

•  the  Life  Cycle  Management  Centers  (LCMCs),  responsible  for  managing 
maintenance  activities  including  logistics,  materiel  and  supply  chain  management, 
and  strategic  sourcing. 

For  instance,  under  the  Aviation  and  Missile  LCMC  (AMCOM),  the  Integrated 
Materiel  Management  Center’s  (IMMC)  Aviation  Logistics  Command  has  offices  for 
Aviation  Fleet  Management  Quality  Assurance  and  a  Quality  Assurance  Surveillance 
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Program.  While  many  organizations  have  responsibility  for  varying  aspects  of  quality,  we 
saw  no  evidence  that  information  regarding  quality  problems  is  communicated  in  a 
coordinated  fashion  to  AMC  leadership. 

The  tools  these  organizations  use  to  track  part  quality  are  heavily  reliant  on  PQDRs. 
Using  PQDR  data  limits  quality  assessment  to  manufacturing  issues,  while  others  like 
maintenance  practices,  part  design  flaws,  training  gaps,  etc.  are  not  captured  by  PQDRs. 
For  these  reasons,  the  Army  manually  examines  logbook  narratives  to  identify  parts  that 
are  being  affected  by  these  other  types  of  quality  issues.  This  manual  process  requires 
significant  SME  resources  that  would  be  better  utilized  in  determining  the  source  of  an 
identified  part  issue  instead  of  compiling  the  data  manually  in  order  to  identify  it. 

The  Army  Needs  an  Enterprise-Level  View  of  Cost  of  Quality 

Currently,  AMC  leadership  has  limited  visibility  of  emerging  or  existing  part  or 
process  quality  problems  that  could  lead  to  diminished  readiness  and  increased  costs. 
Often,  problems  are  brought  to  the  attention  of  leadership  only  after  they  have  reached  a 
critical  juncture.  A  case  in  point  is  a  recent  concern  that  a  bolt  that  is  used  on  a  critical 
component  would  require  replacement  because  of  manufacturing  defects.  While  it  turned 
out  that  this  issue  did  not  require  the  immediate  replacement  of  all  bolts  of  this  type  in  the 
fleet,  it  was  a  reminder  of  how  the  poor  quality  of  even  inexpensive  parts  can  have  severe 
impact  on  costs  and  readiness.  AMC  leadership  was  aware  of  this  problem  because  of  its 
potential  far-reaching  effect  on  the  fleet.  Flowever,  there  are  other  instances  of  part  or 
process  quality  issues  that  may  be  not  be  brought  to  the  attention  of  leadership  because 
their  cost  or  readiness  implications  are  unknown  and  it  is  not  possible  to  judge  the 
severity  of  the  problem. 

Significant  Cost  Avoidance  Is  Attainable 

Poor-quality  parts  and  processes  lead  to  unfavorable  consequences  on  safety, 
readiness,  or  costs.  AMC  management  and  personnel  focus  diligently  on  assuring 
readiness  and  safety.  During  the  last  decade  of  conflict  in  Iraq  and  Afghanistan,  there 
were  strong  pressures  to  increase  production  throughput  at  the  depots  and  the  availability 
of  parts  across  the  services  to  meet  the  surge  in  demands  from  the  field.  In  this 
environment,  the  cost  of  poor  quality  may  have  taken  a  back  seat  to  more  pressing 
demands.  A  briefing  by  the  DCMA  (Swenson,  2009)  states  that  because  of  schedule  and 
mission  requirements,  there  is  pressure  on  DCMA  inspectors  to  accept  nonconforming 
material.  As  a  consequence,  more  parts  enter  the  supply  chain  with  reduced  performance 
and  reliability,  contributing  to  equipment  downtime  and  increased  repair  and  inventory 
costs.  A  study  on  the  effect  of  nonconforming  material  on  manufacturing  lead  times  and 
delivery  schedule  performance  (Nandakumar,  1993)  concludes  that  the  true  cost  of  poor 
quality  may  be  underestimated  if  only  direct  costs  such  as  scrap,  labor,  and  rework  are 
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considered.  The  aforementioned  imply  that  significant  cost  reduction  is  attainable 
through  a  systematic  attempt  to  improve  the  quality  of  process  and  parts. 

As  the  United  States  emerges  from  these  conflicts,  budgetary  pressures  will  require 
the  DoD  to  decrease  their  cost  while  maintaining  readiness  and  assuring  safety. 

Measuring  and  managing  the  cost  of  poor  quality  is  an  essential  part  of  accomplishing 
this  objective. 

A  Pilot  Study  Creating  a  Reporting  Mechanism 

We  recommend  that  AMC  establish  a  pilot  to  test  and  validate  the  tools  presented  in 
this  report  and  to  establish  a  systematic  reporting  of  the  cost  of  poor  quality  across  the 
enterprise. 

Developing  a  prioritized  list  of  potential  part  quality  problems  provides  a  tool  but  not 
a  process  for  its  use.  This  pilot  will  identify  what  is  required  to  integrate  the  tools  and 
methods  for  identifying  poor-quality  parts  and  estimating  their  cost  implications.  As 
mentioned  previously,  many  organizations  are  involved  in  assuring  different  aspects  of 
quality.  AMC  should  define  metrics  that  will  allow  each  organization  to  report  progress 
toward  cost  and  quality  goals  and  systematically  report  this  information  to  AMC 
leadership  in  venues  such  as  the  weekly  production  update  (WPU).  This  will  allow  AMC 
leadership  to  monitor  trends  in  cost  of  poor  quality  over  time  and  help  focus  attention  on 
top  cost  drivers. 

The  cost  of  implementing  will  depend  on  the  cost  of  collecting  the  data,  managing 
the  data,  and  performing  the  analysis.  We  believe  any  incremental  cost  would  be 
outweighed  by  the  potential  benefits.  The  cost  for  data  collection  should  be  nominal, 
since  all  the  necessary  data  is  already  available.  Additional  costs  for  extracting  the 
required  data  and  maintaining  the  required  database  would  depend  on  whether  this  tool 
can  be  incorporated  into  an  already  existing  data  analysis  framework  or  if  a  new  one 
needs  to  be  created.  Because  most  of  the  analysis  is  automated,  the  burden  on  the  analysts 
is  reduced;  the  root  cause  analysis  that  is  already  performed  would  be  redirected  using  a 
prioritized  list  of  parts  created  by  the  method. 

A  potential  weakness  of  this  method  and  one  that  would  undermine  its  usefulness  is 
an  overabundance  of  false  indications  of  part  quality  problems.  We  have  sought  to 
mitigate  this  risk  by  imposing  multiple  conservative  criteria  that  must  be  met  before  we 
identify  a  part  as  a  potential  problem.  However,  even  so  there  is  a  chance  that  the  root 
cause  of  a  problem  may  be  difficult  to  ascertain,  thus  leading  to  a  frustration  on  the  part 
of  the  analysts  and  reduced  confidence  in  the  tool.  The  pilot  test  should  closely  monitor 
this  risk  and  attempt  to  calibrate  the  results  of  the  analysis  to  minimize  it. 
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Appendix  A.  Crow-AMSAA  Method 


Depending  upon  the  reliability  of  changes  in  the  fleet  that  would  cause  the  reliability 
to  change  (e.g.,  new  maintenance  procedures,  part  improvement,  training,  etc.),  the 
MTBF  of  parts  in  the  Army  fleet  will  improve  or  worsen  at  different  rates. 

The  Crow-AMSAA  method  is  based  on  a  discovery  made  in  the  1960s  concerning  the 
relationship  between  cumulative  fleet  time  and  cumulative  fleet  failures,  which  is  known 
as  the  Duane  postulate.  This  postulate  states  that  the  relationship  between  cumulative 
fleet  time  and  cumulative  fleet  failures  is  often  linear  on  a  log-log  scale  (Crow,  2010). 

The  Crow-ASMAA  method  builds  upon  the  Duane  postulate  by  calculating  the  expected 
number  of  failures: 


E[N(t)]  =  Xtp ,  (A.l) 

where  N(t)  is  the  expected  number  of  failures  up  to  a  fleet  time  t,  X  is  the  scale  parameter, 
and  /?  is  the  slope  on  the  log-log  plot. 

logfs[iV(f)]  =  log  A  +  /31ogf  (A.2) 

The  parameters  X36  and  [1  can  be  estimated  by  using  several  linear  regression  techniques 
including  least  squares,  maximum  likelihood  estimators  (MLE),  or  generalized  linear 
models  (GLM). 

In  this  analysis,  we  use  the  MLE  to  calculate  beta  and  lambda;  see  equations  (A. 3) 
and  (A. 4)  respectively. 
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The  [:>  in  the  Crow-ASMAA  method  has  a  similar  interpretation  as  the  Weibull  [>.  When  [:> 
is  less  than  one,  the  failure  inter-arrival  time  is  increasing — fleet  failures  are  happening 
less  frequently  as  fleet  time  increases,  which  means  the  MTBF  is  improving  with  time. 

36  When  P  =1,  Xt  represents  the  number  of  failures  at  time  t  for  a  system  with  a  stable  failure  rate  over  time. 
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When  p  is  equal  to  one,  the  failure  inter-arrival  time  is  neither  increasing  nor  decreasing 
with  time — the  MTBF  remains  constant.  When  / 3  is  greater  than  one,  the  failure  inter¬ 
arrival  time  is  decreasing — the  MTBF  is  worsening  across  time. 

Significant  changes  in  the  fleet  will  cause  the  [i  and  2  values  to  change  in  (A.  1)  and 
introduce  new  linear  regions  on  the  cumulative  failures  versus  cumulative  flight  hours 
log-log  plot. 

For  example,  Sun  et  al.  (2005)  used  the  Crow-AMSAA  method  to  monitor  the 
reliability  of  the  condensate  air  removal  system  pumps  in  a  nuclear  power  plant.  The 
plotted  data  on  the  log-log  plot  had  two  distinct  linear  regions.  The  first  region  had  a  /? 
value  less  than  one — MTBF  was  improving — and  the  most  recent  linear  region  had  a  [> 
greater  than  one — MTBF  was  worsening.  This  change  in  the  ft  value  alerted  system 
engineering37  that  there  was  an  issue  and  during  the  subsequent  investigation,  it  was 
discovered  that  moisture  entering  one  of  the  bearing  assemblies  was  the  cause.  Abernethy 
(2009)  used  the  Crow-AMSAA  approach  to  assess  how  an  in-service  maintenance 
change  affected  the  number  of  failures  seen  in  a  system.  The  second  linear  region  in  this 
log-log  plot  had  a  [i  value  less  than  one,  indicating  that  this  maintenance  change  was 
improving  the  system  reliability. 

These  examples  illustrate  that  significant  changes  that  affect  part  reliability 
performance  can  be  identified  by  examining  failure  data  on  a  log-log  plot  where  the 
x-axis  is  the  cumulative  operating  time  and  the  y-axis  is  the  cumulative  failures.  Also, 
fitting  (A.  1)  to  these  linear  regions  will  estimate  the  [>  value,  which  is  used  to  quantify  the 
effect  of  the  change. 

The  Crow-AMSAA  method  relies  on  a  few  assumptions  and  is  considered  best 
practice  among  other  similar  reliability  techniques.  To  properly  use  the  Crow-AMSAA 
method,  the  failure  data  must  be  linear  on  a  log-log  plot  and  there  must  be  at  least  five 
failure  points  (Abernethy,  2009).  Also,  if  the  Crow-ASMAA  method  is  used  to 
distinguish  between  different  linear  regions  due  to  a  reliability  improvement  in  the  fuel 
pump,  aircraft  fuel  pump  failures  should  be  random.  Many  facets  of  the  Crow-AMSAA 
method  have  been  analyzed.  Overall,  this  method  is  considered  best  practice  in  its  realm 
of  application  (Abernethy,  2009):  an  Air  Force  study  by  McGlone  (1984)  found  that  the 
Crow-AMSAA  method  has  the  most  accurate  forecasts;  the  U.S.  defense  standard  MIL- 
HDBK-189  states  that  the  Crow-ASMAA  method  is  the  best  for  reliability  growth 
management;  and  Wang  and  Coit  (2005)  concluded  that  the  Crow-AMSAA  method  is 
best  for  identifying  changes  in  reliability  trends. 


37 

It  took  system  engineering  nine  months  to  realize  that  there  was  a  problem  with  this  pump  from  using 
the  Crow-AMSAA  method.  One  reason  for  this  delay  is  that  it  takes  at  least  five  failures  to  fit  a  new  linear 
region  with  this  technique  (Abernethy,  2009). 
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Appendix  B.  Clustering  Like  Parts 


Cluster  analysis  aims  to  both  combine  parts  with  similar  features  and  separate  parts 
with  differing  features,  thus  producing  rather  homogenous  groupings  of  parts  known  as 
clusters.  Clustering  is  an  unsupervised  method  that  relies  upon  a  feature  set  to  construct  a 
similarity  measure  that  is  then  used  to  identify  structure  within  the  data.  Clustering  is 
used  across  many  fields  such  as  information  technology,  biology,  artificial  intelligence, 
and  marketing  (Gan,  Ma,  and  Wu,  2007).  To  appropriately  allocate  this  method  for  part 
reliability,  we  apply  a  hard  clustering  algorithm  known  as  hierarchical  clustering  or 
connectivity-based  clustering,  which  groups  similar  U.S.  Army  parts.  These  groupings 
allow  us  to  make  comparisons  of  the  demand  streams  of  all  parts  within  a  cluster. 

The  U.S.  Army  classifies  all  manufactured  parts  under  a  Federal  Supply  Class  (FSC). 
Each  FSC  category  broadly  identifies  a  set  or  type  of  parts.  All  parts  within  an  FSC 
comprise  a  dataset,  X,  that  is  Mby  M,  where  M  is  the  number  of  parts  contained  in  the 
particular  FSC.  The  clustering  goal  is  to  assign  a  group  index,/?,  to  each  part,  x*,  where 
each  part,  x*  e  X,  belongs  to  one  cluster,  Cp.  Within  an  FSC  the  parts  can  be  thought  of  as 
hierarchy,  that  is,  all  nested  under  one  FSC.  This  data  structure  lends  itself  nicely  to  the 
hierarchical  clustering  methods.  The  end  result  of  the  clustering  within  an  FSC  is  a 
hierarchical  representation  where  at  the  leaves  (bottom)  there  is  one  part  (a  single 
observation)  and  the  top  (the  entire  cluster)  contains  all  the  data  for  the  FSC. 

We  use  hierarchical  clustering  to  divide  parts  within  each  FSC  into  a  sequence  of 
nested  partitions.  This  is  an  agglomerative  (bottom-up)  clustering  algorithm  that  starts 
with  every  part  in  its  own  cluster.  Next  the  pairwise-cluster  similarities  are  computed, 
and  then  the  most  similar  clusters  are  repetitively  merged  based  upon  a  geometric 
similarity  criterion,  generally  known  as  a  distance.  Clusters  are  repetitively  merged  until 
the  preferred  number  of  clusters  is  achieved  (this  is  discussed  later).  The  decision  to  use 
the  agglomerative  hierarchical  method,  was  based  on  a  comparison  of  internal  measures 
(i.e.,  compactness,  connectedness,  and  separation)  in  the  clusters  for  many  types  of 
clustering  algorithms,  such  as  k-means,  divisive  hierarchical,  partitioning  around 
medoids,  and  fuzzy  clustering.  We  characterize  the  similarity  and  perform  the 
partitioning  of  the  data  through  the  application  of  a  minimum  variance  method  known  as 
Ward’s  method  (Ward,  1963,  and  Ward  and  Hook,  1963),  where  the  objective  is  to 
minimize  the  loss  of  infonnation  at  each  stage  of  partitioning.  Ward’s  method 
characterizes  the  dissimilarity,  Dtj ,  between  two  clusters,  C)  and  Cj,  by  the  following 
equation: 
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where  Q  and  Cj  are  the  mean  vectors  for  cluster  i  and  j  respectively,  and  A,  and  Nj  are 
the  the  number  of  observations  in  each  cluster.  Ward’s  method  tends  to  produce  clusters 
of  approximately  equal  size  and  is  also  sensitive  to  outliers  (Milligan  and  Sokol,  1980). 
While  there  are  many  different  types  of  metrics  we  could  have  used  (e.g.,  single-link, 
complete-link,  or  group  average  are  all  graph  methods),  we  chose  a  geometric  method 
because  this  metric  subjectively  appeared  to  produce  clusters  that  were  most  sensible  to 
the  Army.  This  geometric  method  means  that  parts  within  a  cluster  can  be  represented  by 
a  center  point  (Gan,  Ma,  and  Wu,  2007). 

To  construct  the  dissimilarity  matrix,  from  which  the  pairwise  similarities  are 
computed,  we  use  a  feature  set  consisting  of  a  maximum  of  six  distinguishing  part 
features:  nomenclature,  price,  weight,  and  part  dimensions  (height,  width,  and  cube  size). 
Each  feature  is  standardized  on  a  zero  to  one  scale.  A  dissimilarity  matrix  is  then 
generated  using  the  Gower  general  coefficient  (Gower,  1971). 18  Weights  are  assigned  to 
each  variable  where  the  weights  sum  to  unity — price  is  weighted  at  0.5,  nomenclature  at 
0.25,  and  weight  and  part  dimension  at  0.0625  each.  Under  this  weighting  scheme,  price 
is  the  primary  driver  of  similarity  among  the  part  clusters,  followed  by  nomenclature.  If 
weight  or  part  dimensions  do  not  explain  significant  variation  in  the  data,  they  are 
dropped  from  the  feature  set  and  their  variable  weight  is  reallocated  to  price  and 
nomenclature.  Significant  variation  is  determined  from  a  principal  components  analysis 
by  examining  the  variable  loadings  from  components  that  account  for  80  percent  of  the 
data’s  variation. 

Following  the  construction  of  the  dissimilarity  matrix,  the  appropriate  number  of 
clusters  within  each  FSC  is  determined  and  is  used  as  the  stopping  criterion  for  the 
hierarchical  clustering  algorithm.  While  numerous  statistics  have  been  proposed  to  decide 
upon  the  optimal  cluster  size  (Gentle,  2002),  we  maximize  the  within-cluster  variance  by 
using  the  Calinski-Harabasz  (CH)  Index  (Calinski  and  Harabasz,  1974): 


CH  (k)  = 


b/ 

/(k- 1) 
w/ 

/( n  —  k ) 


(B.2) 
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Future  research  will  use  additional  variables  that  may  by  ordinal,  categorical,  or  binary.  The  Gower 
metric  includes  various  distance  measures  for  different  types  of  variables.  For  numeric,  continuous 
variables  in  Gower,  the  Manhattan  distance  is  used  (Romesburg,  1984). 
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where  k  is  the  number  of  clusters,  n  is  the  maximum  number  of  clusters  allowed,  which 
we  set  to  (M-  l)/4,  b  is  the  between-cluster  sum  of  squares,  and  w  is  the  within-cluster 
sum  of  squares.  The  value  of  k  that  maximizes  CH  is  the  number  of  clusters  we  use. 
Milligan  and  Cooper  (1985)  performed  a  Monte  Carlo  study  comparing  30  different 
cluster  validation  measures  and  detennined  that  the  CH  Index  was  the  superior 
performer.  With  the  number  of  clusters  known,  we  restrict  the  hierarchical  cluster 
algorithm  to  this  number,  then  apply  the  algorithm  to  the  dissimilarity  matrix  and  obtain 
the  groupings  within  each  FSC. 

The  next  step  involves  comparing  the  CTASC  demands  of  each  part  within  a  cluster. 
For  each  part  we  create  a  time-series  by  standardizing  monthly  CTASC  demands  by  the 
monthly  aircraft  hours  flown.  The  standardized  CTASC  time-series  for  all  parts  within  a 
cluster  are  then  plotted.  Since  the  features  of  the  parts  within  a  cluster  are  similar,  we 
hypothesize  that  the  parts  are  also  similar  and  as  such,  the  standardized  demands  should 
also  be  similar.  If  a  part  exhibits  unusually  high  demand  relative  to  the  rest  of  the  parts 
within  the  cluster,  this  part  would  warrant  further  investigation.  We  use  the 
nonparametric  Wilcoxon-Mann- Whitney  hypothesis  test  to  compare  one  series  to  the 
median  of  all  other  series.  This  is  a  widely  used  statistical  technique  to  determine  whether 
one  of  two  samples  is  statistically  larger  than  another.  By  using  a  nonparametric  test  we 
are  able  to  overcome  the  distributional  assumptions  inherent  in  parametric  hypothesis 
tests.  Additionally,  with  this  particular  test  the  samples  can  have  unequal  observations. 
This  test  does  assume  that  the  observations  are  independent  and  that  they  are  numeric. 
Since  the  data  is  time-series  in  nature,  it  is  not  truly  independent,  though  we  did  not  find 
strong  serial  correlation  in  many  of  the  standardized  demand  streams  we  inspected. 
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